pi-agent-browser-native 0.2.68 → 0.2.70
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +36 -0
- package/README.md +11 -2
- package/dist/extensions/agent-browser/index.js +3 -4
- package/dist/extensions/agent-browser/lib/command-taxonomy.js +5 -1
- package/dist/extensions/agent-browser/lib/config-policy.js +3 -24
- package/dist/extensions/agent-browser/lib/config.js +0 -14
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/wait-timeouts.js +80 -26
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare.js +12 -4
- package/dist/extensions/agent-browser/lib/playbook.js +3 -10
- package/dist/extensions/agent-browser/lib/results/editable-ref-evidence.js +0 -3
- package/dist/extensions/agent-browser/lib/results/presentation/registry.js +4 -0
- package/dist/extensions/agent-browser/lib/runtime.js +0 -1
- package/docs/ARCHITECTURE.md +2 -2
- package/docs/COMMAND_REFERENCE.md +39 -5
- package/docs/ELECTRON.md +2 -2
- package/docs/RELEASE.md +3 -3
- package/docs/REQUIREMENTS.md +2 -7
- package/docs/SUPPORT_MATRIX.md +15 -14
- package/docs/TOOL_CONTRACT.md +6 -5
- package/package.json +1 -2
- package/platform-smoke.config.mjs +2 -3
- package/scripts/agent-browser-capability-baseline.mjs +22 -3
- package/scripts/platform-smoke.mjs +1 -2
- package/dist/extensions/agent-browser/lib/orchestration/browser-run.js +0 -1
- package/dist/extensions/agent-browser/lib/results/shared.js +0 -20
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,41 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.70 - 2026-07-18
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- Removed unused TypeScript imports, locals, exports, declaration sidecars, compatibility barrels, inert `browser.defaultLaunchArgs` config handling, and duplicated platform/build orchestration while preserving the browser runtime contract.
|
|
8
|
+
- Condensed completed plans and the historical support ledger into current ADR/decision indexes, removed obsolete archive material, and enabled `noUnusedLocals` for ongoing enforcement.
|
|
9
|
+
- Kept one guaranteed build owner before every `dist/` consumer; package verification now lets `prepare` create a missing `dist/` before expanding the publish contract and forbids internal `docs/plans/` content by prefix.
|
|
10
|
+
|
|
11
|
+
### Compatibility
|
|
12
|
+
|
|
13
|
+
- Undocumented deep imports through `lib/results/shared.js` or `lib/orchestration/browser-run.js`, the inert `browser.defaultLaunchArgs` config field, unused named platform-config exports, and the redundant default-export `supportedTargets` platform-smoke property are no longer supported. Native `agent_browser` behavior and documented package entrypoints are unchanged.
|
|
14
|
+
|
|
15
|
+
### Validation
|
|
16
|
+
|
|
17
|
+
- Passed `npm run verify -- release` (587 tests passed, 2 opt-in skips), configured-source lifecycle, packaged Pi smoke, live command-reference verification, and macOS/Ubuntu/native-Windows Crabbox `platform-build` plus `browser-dogfood-smoke` suites.
|
|
18
|
+
- Passed opt-in real-upstream, deterministic dogfood, benchmark, and startup-profile gates; the three-sample startup maximum was 51.9 ms against the 250 ms budget.
|
|
19
|
+
- Passed a checkout-loaded Pi 0.80.10 tmux smoke on `example.com` and `react.dev`, including QA, fresh-session snapshot/link navigation, verified screenshot evidence, recovery from stale selector attempts, and zero active browser sessions after cleanup.
|
|
20
|
+
|
|
21
|
+
## 0.2.69 - 2026-07-17
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
|
|
25
|
+
- Rebaselined the command/help inventory, source evidence, real-upstream output-shape fixture, and package docs to `agent-browser 0.32.2` / vercel-labs/agent-browser@6ede7a9470ac4b681cabf838af8668b9aa99e957.
|
|
26
|
+
- Documented the 0.32.1–0.32.2 eve compatibility, packaging, stable AI SDK, and scoped-config updates without adding an eve-specific Pi runtime or dependency.
|
|
27
|
+
- Added the previously missing upstream `read [url]` surface to the local capability inventory and prompt guidance, including markdown/llms/outline/filter options and explicit long-timeout budgeting across upstream's per-request fallback sequence.
|
|
28
|
+
|
|
29
|
+
### Fixed
|
|
30
|
+
|
|
31
|
+
- Rendered successful `read` results from upstream `data.content` instead of collapsing them to the fetched URL.
|
|
32
|
+
- Kept explicit `read <url>` metadata from replacing the active browser tab target used by later ref and tab recovery.
|
|
33
|
+
|
|
34
|
+
### Validation
|
|
35
|
+
|
|
36
|
+
- Passed `npm run verify` (585 tests passed, 2 opt-in skips), live command-reference verification, and `npm run verify -- real-upstream` (2/2 tests) against installed `agent-browser 0.32.2`.
|
|
37
|
+
- Passed a checkout-loaded Pi 0.80.10 tmux smoke: `read https://example.com` rendered the full `Example Domain` body and the managed session closed cleanly.
|
|
38
|
+
|
|
3
39
|
## 0.2.68 - 2026-07-16
|
|
4
40
|
|
|
5
41
|
### Changed
|
package/README.md
CHANGED
|
@@ -78,7 +78,7 @@ The result is optimized for agent work:
|
|
|
78
78
|
| Auth/profile workflows can leak secrets | Supports `auth save --password-stdin` and redacts sensitive args, URLs, stdout/stderr, details, and parse-failure spills | `test/agent-browser.extension-security-redaction.test.ts` |
|
|
79
79
|
| Stateful cookies/storage/auth output bloats or leaks context | Presentation layer redacts `details.data` for cookies and credential-like storage values while keeping low-risk local QA values such as `theme: dark` readable; recursively scrubs other structured upstream JSON (network, diff, trace/profiler, stream, dashboard, chat, auth, dialog, frame, state, and similar) using sensitive key names plus string heuristics; masks sensitive argv flags and positionals; scrubs secrets from failed batch step errors; and exposes a compact redacted `batch` matrix on top-level `details.data` | `extensions/agent-browser/lib/results/presentation.ts`, `extensions/agent-browser/lib/results/presentation/diagnostics.ts`, `extensions/agent-browser/lib/runtime.ts`, `test/agent-browser.presentation-diagnostics.test.ts` |
|
|
80
80
|
| Stale `@eN` refs fail mysteriously | Records per-session `details.refSnapshot`, rejects mismatched URLs / unknown refs / unsafe `batch` stdin ordering before spawn, adds recovery guidance to rerun `snapshot -i` or use stable `find` locators | `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/session-page-state.ts`, `test/agent-browser.session-page-state.test.ts`, `test/agent-browser.results.test.ts`, `test/agent-browser.extension-ref-guards.test.ts`, `test/agent-browser.extension-semantic-recovery.test.ts` |
|
|
81
|
-
| Agents need stable success/failure buckets | Exposes bounded `resultCategory`, `successCategory`, and `failureCategory` on tool `details` for branching without parsing prose; a `tool_result` hook also aligns real Pi `isError` semantics, naming `Pi tool isError: true` in prose output while preserving parseable caller-requested `--json` output | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/results/categories.ts`, `extensions/agent-browser/
|
|
81
|
+
| Agents need stable success/failure buckets | Exposes bounded `resultCategory`, `successCategory`, and `failureCategory` on tool `details` for branching without parsing prose; a `tool_result` hook also aligns real Pi `isError` semantics, naming `Pi tool isError: true` in prose output while preserving parseable caller-requested `--json` output | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/results/categories.ts`, `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/pi-tool-rendering.ts`, `test/agent-browser.results.test.ts`, `test/agent-browser.extension-validation.test.ts`, `test/agent-browser.pi-pipeline.test.ts` |
|
|
82
82
|
| Clicks can report success without the page receiving the event | Top-level non-Electron direct `click` calls on `xpath=` targets or role-gated current `@e…` refs (`button`, `checkbox`, `menuitem`, `radio`, `switch`, `tab`) install a bounded target-specific DOM-event probe; eligible `@e…` refs use the latest snapshot role/name metadata, and duplicate-name refs use snapshot-order `duplicateIndex` rather than requiring a unique name. If upstream reports success but no trusted event reaches the resolved target, the wrapper fails the tool, exposes `details.clickDispatch`, and suggests explicit retry/inspect next actions (no in-page replay), including a nested-scroll `scrollintoview` action when the probe sees the target outside a scroll container or viewport. Unresolved locator clicks such as raw `find … click` are left upstream-owned to avoid false failures for frame-scoped targets. Other click results still expose `details.pageChangeSummary`, and unchanged-URL clicks can surface evidence-backed `details.overlayBlockers` candidates. | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/orchestration/browser-run/click-dispatch.ts`, `extensions/agent-browser/lib/results/presentation/navigation.ts`, `test/agent-browser.presentation.test.ts`, `test/agent-browser.extension-click-dispatch.test.ts` |
|
|
83
83
|
| Dashboard scroll commands can look successful while nothing moves | Samples viewport and prominent scroll-container positions around large top-level `scroll` calls on an existing or fresh managed session; unchanged positions produce `details.scrollNoop`, visible recovery guidance, and exact `nextActions` for snapshot/screenshot verification | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#core-page-and-element-commands), `test/agent-browser.extension-validation.test.ts` |
|
|
84
84
|
| Dropdown/combobox clicks can focus or hit native option box-model errors | Adds first-class `select <selector> <value...>` paths through raw `args`, `semanticAction`, and `job`; for custom combobox clicks, detects focused controls with explicit `aria-expanded` state but no visible options and returns `details.comboboxFocus` plus exact recovery `nextActions` | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#core-page-and-element-commands), `extensions/agent-browser/lib/input-modes/semantic-action.ts`, `test/agent-browser.extension-input-modes.test.ts`, `test/agent-browser.extension-validation.test.ts` |
|
|
@@ -302,6 +302,15 @@ Run a multi-step flow in one tool call:
|
|
|
302
302
|
|
|
303
303
|
If the same `batch` stdin later uses `@e…` on interaction commands after a step that can navigate or mutate the page (`open`, non-form `click`, `reload`, and similar), insert a `snapshot` step whose first argv token is `snapshot` (for example `["snapshot","-i"]`) between those phases. Multiple same-snapshot `fill @e…` steps and native form-control steps (`check`/`uncheck` on checkbox or radio refs, checkbox/radio `click`/`tap` refs, and `select` on combobox refs) may be batched before a final click/submit step. Dynamic or autosubmit forms should still use stable locators or split with a fresh snapshot. The wrapper rejects unsafe ordering with `failureCategory: "stale-ref"` before upstream runs; full rules are under `refSnapshot` in [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details).
|
|
304
304
|
|
|
305
|
+
Read documentation or other unstructured text without launching Chrome, or omit the URL to read the rendered DOM of the current tab:
|
|
306
|
+
|
|
307
|
+
```json
|
|
308
|
+
{ "args": ["read", "https://example.com/docs", "--filter", "authentication"] }
|
|
309
|
+
{ "args": ["read"] }
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
Explicit URL reads prefer `text/markdown`, then try a `.md` path and nearby `llms.txt` links before falling back to readable HTML text. Use `--outline`, `--llms index|full`, `--require-md`, `--raw`, or `--timeout <ms>` when needed. The wrapper renders upstream `data.content` first, preserves metadata in `details.data`, keeps fetched URLs from replacing the active browser tab target, and budgets explicit long read timeouts across upstream's `.md` and ancestor-`llms.txt` request fallbacks.
|
|
313
|
+
|
|
305
314
|
Evaluate page JavaScript through stdin. Put the script in the top-level `stdin` field, not as an extra `args` token after `--stdin`. Return the value you want as an expression; `eval --stdin` may warn with `details.evalStdinHint` when a function-shaped snippet serializes to `{}` instead of being invoked:
|
|
306
315
|
|
|
307
316
|
```json
|
|
@@ -592,7 +601,7 @@ For larger local handoffs or PR-ready confidence before expensive release/lifecy
|
|
|
592
601
|
npm run verify -- pre-pr
|
|
593
602
|
```
|
|
594
603
|
|
|
595
|
-
That mode composes the full default gate with `npm run verify -- package`, so package contents and forbidden
|
|
604
|
+
That mode composes the full default gate with `npm run verify -- package`, so package contents and forbidden repo-only files are checked without launching Pi lifecycle, Crabbox, or live dogfood flows. Package modes build through npm `prepare`; lifecycle and startup-profile build in their focused scripts; default and platform-target build before consuming `dist/`, so clean checkouts do not validate stale or missing compiled output. The same `prepare` script owns GitHub/source installs; when Pi installs with `npm install --omit=dev`, it installs the source-build dev dependencies with lifecycle scripts disabled before building the ignored `dist/` entrypoint that Pi loads.
|
|
596
605
|
|
|
597
606
|
The deterministic agent-efficiency benchmark’s **standalone JSON/Markdown accounting run** is not part of default or pre-PR `npm run verify` (only `npm run verify -- benchmark` or `npm run benchmark:agent-browser` invokes the script). The full unit suite still exercises `test/agent-browser.efficiency-benchmark.test.ts`. Use the script before and after agent-facing abstractions to prove call-count, output-size, stale-ref, artifact, failure-category coverage, success-rate, and elapsed-time effects before changing the wrapper UX:
|
|
598
607
|
|
|
@@ -15,21 +15,20 @@ import { createEphemeralSessionSeed, createFreshSessionName, createImplicitSessi
|
|
|
15
15
|
import { isRecord } from "./lib/parsing.js";
|
|
16
16
|
import { buildPromptPolicy, getLatestUserPrompt, shouldAppendBrowserSystemPrompt } from "./lib/prompt-policy.js";
|
|
17
17
|
import { isCloseCommand } from "./lib/command-taxonomy.js";
|
|
18
|
-
import { cleanupSecureTempArtifacts
|
|
18
|
+
import { cleanupSecureTempArtifacts } from "./lib/temp.js";
|
|
19
19
|
import { AGENT_BROWSER_PARAMS, } from "./lib/input-modes.js";
|
|
20
20
|
import { parseAllowedDomainsPolicyFromArgs } from "./lib/navigation-policy.js";
|
|
21
|
-
import { closeManagedSession, getSessionContextKey, runAgentBrowserTool } from "./lib/orchestration/browser-run.js";
|
|
21
|
+
import { closeManagedSession, getSessionContextKey, runAgentBrowserTool } from "./lib/orchestration/browser-run/index.js";
|
|
22
22
|
import { findElectronLaunchRecordForSession, getActiveElectronRecords } from "./lib/orchestration/browser-run/session-state.js";
|
|
23
23
|
import { parseBatchStdinJsonArray } from "./lib/orchestration/batch-stdin.js";
|
|
24
24
|
import { ELECTRON_POST_COMMAND_STATUS_SETTLE_MS, ELECTRON_PROFILE_ISOLATION_DETAILS, cleanupActiveElectronHostLaunches, handleElectronHostInput, restoreElectronLaunchRecordsFromBranch, } from "./lib/orchestration/electron-host/index.js";
|
|
25
25
|
import { buildValidationFailureResult, resolveAgentBrowserInput } from "./lib/orchestration/input-plan.js";
|
|
26
26
|
import { applyAgentBrowserOutputPath } from "./lib/orchestration/output-file.js";
|
|
27
|
-
import { isSessionArtifactManifest
|
|
27
|
+
import { isSessionArtifactManifest } from "./lib/results/artifact-manifest.js";
|
|
28
28
|
import { canRegisterWebSearchTool, loadAgentBrowserConfigSync } from "./lib/config.js";
|
|
29
29
|
import { createAgentBrowserWebSearchTool } from "./lib/web-search.js";
|
|
30
30
|
import { isDirectAgentBrowserBashAllowed, isHarmlessAgentBrowserInspectionCommand, looksLikeDirectAgentBrowserBash, } from "./lib/bash-guard.js";
|
|
31
31
|
import { AgentBrowserResultComponent, buildAgentBrowserToolResultPatch, formatAgentBrowserRenderCall, formatAgentBrowserRenderResult, } from "./lib/pi-tool-rendering.js";
|
|
32
|
-
const DEFAULT_SESSION_MODE = "auto";
|
|
33
32
|
function isBashToolCallEvent(event) {
|
|
34
33
|
if (!isRecord(event) || event.toolName !== "bash" || !isRecord(event.input))
|
|
35
34
|
return false;
|
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
* Scope: Static command capability taxonomy only; command-shape parsing, spawning, and formatting live elsewhere.
|
|
6
6
|
*/
|
|
7
7
|
const ADDITIONAL_COMMAND_TOKENS = [
|
|
8
|
-
"auth", "chat", "clipboard", "confirm", "connect", "dashboard", "deny", "device", "dialog", "diff", "doctor", "errors", "eval", "find", "frame", "get", "highlight", "inspect", "install", "is", "mcp", "plugin", "plugins", "profiles", "profiler", "react", "
|
|
8
|
+
"auth", "chat", "clipboard", "confirm", "connect", "dashboard", "deny", "device", "dialog", "diff", "doctor", "errors", "eval", "find", "frame", "get", "highlight", "inspect", "install", "is", "mcp", "plugin", "plugins", "profiles", "profiler", "react", "record", "removeinitscript", "session", "set", "skills", "snapshot", "state", "stream", "trace", "upgrade", "vitals", "wait", "web-vitals", "window",
|
|
9
9
|
];
|
|
10
10
|
const COMMAND_CAPABILITIES = [
|
|
11
11
|
{
|
|
@@ -178,6 +178,10 @@ const COMMAND_CAPABILITIES = [
|
|
|
178
178
|
navigationObservable: true,
|
|
179
179
|
triggersPostMutationSnapshot: true,
|
|
180
180
|
},
|
|
181
|
+
{
|
|
182
|
+
command: "read",
|
|
183
|
+
readOnlyDiagnosticSessionTarget: true,
|
|
184
|
+
},
|
|
181
185
|
{
|
|
182
186
|
command: "screenshot",
|
|
183
187
|
eligibleForPageChangeSummary: true,
|
|
@@ -16,7 +16,7 @@ import { join, resolve } from "node:path";
|
|
|
16
16
|
/** @typedef {{ provider: WebSearchProvider; apiKeyEnv: string; configKey: WebSearchProviderConfigKey; label: string }} WebSearchProviderDescriptor */
|
|
17
17
|
/** @typedef {{ name: string; policy?: BrowserDefaultProfilePolicy }} BrowserDefaultProfileConfig */
|
|
18
18
|
/** @typedef {{ enabled?: boolean; preferredProvider?: WebSearchProvider; braveApiKey?: string; exaApiKey?: string }} WebSearchConfig */
|
|
19
|
-
/** @typedef {{ defaultProfile?: BrowserDefaultProfileConfig; executablePath?: string
|
|
19
|
+
/** @typedef {{ defaultProfile?: BrowserDefaultProfileConfig; executablePath?: string }} BrowserConfig */
|
|
20
20
|
/** @typedef {{ version?: 1; webSearch?: WebSearchConfig; browser?: BrowserConfig }} AgentBrowserConfig */
|
|
21
21
|
/** @typedef {{ config: AgentBrowserConfig; path: string; scope: ConfigLayerScope }} ConfigLayer */
|
|
22
22
|
/** @typedef {{ kind: CredentialSourceKind; provider?: WebSearchProvider; rawValue: string; scope: AgentBrowserConfigScope }} CredentialSource */
|
|
@@ -155,21 +155,6 @@ function validateString(value, path, errors) {
|
|
|
155
155
|
}
|
|
156
156
|
return value;
|
|
157
157
|
}
|
|
158
|
-
/**
|
|
159
|
-
* @param {unknown} value
|
|
160
|
-
* @param {string} path
|
|
161
|
-
* @param {string[]} errors
|
|
162
|
-
* @returns {string[] | undefined}
|
|
163
|
-
*/
|
|
164
|
-
function validateStringArray(value, path, errors) {
|
|
165
|
-
if (value === undefined)
|
|
166
|
-
return undefined;
|
|
167
|
-
if (!Array.isArray(value) || value.some((entry) => typeof entry !== "string")) {
|
|
168
|
-
errors.push(`${path} must be an array of strings.`);
|
|
169
|
-
return undefined;
|
|
170
|
-
}
|
|
171
|
-
return value;
|
|
172
|
-
}
|
|
173
158
|
/**
|
|
174
159
|
* @param {unknown} value
|
|
175
160
|
* @param {string} path
|
|
@@ -244,12 +229,11 @@ export function isProjectSafeCredentialValueForProvider(rawValue, provider) {
|
|
|
244
229
|
/**
|
|
245
230
|
* @param {unknown} value
|
|
246
231
|
* @param {string} path
|
|
247
|
-
* @param {ConfigLayerScope} scope
|
|
248
232
|
* @param {string[]} errors
|
|
249
233
|
* @param {string[]} warnings
|
|
250
234
|
* @returns {AgentBrowserConfig | undefined}
|
|
251
235
|
*/
|
|
252
|
-
export function validateAgentBrowserConfig(value, path,
|
|
236
|
+
export function validateAgentBrowserConfig(value, path, errors, warnings) {
|
|
253
237
|
if (!isRecord(value)) {
|
|
254
238
|
errors.push(`${path} must contain a JSON object.`);
|
|
255
239
|
return undefined;
|
|
@@ -297,11 +281,6 @@ export function validateAgentBrowserConfig(value, path, scope, errors, warnings)
|
|
|
297
281
|
if (executablePath) {
|
|
298
282
|
config.browser.executablePath = executablePath;
|
|
299
283
|
}
|
|
300
|
-
const defaultLaunchArgs = validateStringArray(value.browser.defaultLaunchArgs, `${path}.browser.defaultLaunchArgs`, errors);
|
|
301
|
-
if (defaultLaunchArgs) {
|
|
302
|
-
config.browser.defaultLaunchArgs = defaultLaunchArgs;
|
|
303
|
-
warnings.push(`${path}.browser.defaultLaunchArgs is recorded for future use; current releases do not auto-inject default launch args.`);
|
|
304
|
-
}
|
|
305
284
|
}
|
|
306
285
|
}
|
|
307
286
|
for (const key of Object.keys(value)) {
|
|
@@ -328,7 +307,7 @@ export function parseAgentBrowserConfigLayer(raw, path, scope, errors, warnings)
|
|
|
328
307
|
errors.push(`Could not parse ${scope} config ${path}: ${error instanceof Error ? error.message : String(error)}`);
|
|
329
308
|
return undefined;
|
|
330
309
|
}
|
|
331
|
-
const config = validateAgentBrowserConfig(parsed, path,
|
|
310
|
+
const config = validateAgentBrowserConfig(parsed, path, errors, warnings);
|
|
332
311
|
return config ? { config, path, scope } : undefined;
|
|
333
312
|
}
|
|
334
313
|
/**
|
|
@@ -106,17 +106,3 @@ export async function resolvePreferredWebSearchCredential(state, options = {}) {
|
|
|
106
106
|
}
|
|
107
107
|
return undefined;
|
|
108
108
|
}
|
|
109
|
-
export async function hasResolvableCredentialSource(state, options = {}) {
|
|
110
|
-
if (!state.webSearchEnabled || state.errors.length > 0)
|
|
111
|
-
return false;
|
|
112
|
-
for (const provider of getWebSearchProviderOrder(state)) {
|
|
113
|
-
const source = getWebSearchCredentialSource(state, provider);
|
|
114
|
-
if (!source)
|
|
115
|
-
continue;
|
|
116
|
-
if (source.kind === "command")
|
|
117
|
-
return true;
|
|
118
|
-
if ((await resolveCredentialSource(source, options))?.value)
|
|
119
|
-
return true;
|
|
120
|
-
}
|
|
121
|
-
return false;
|
|
122
|
-
}
|
package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/wait-timeouts.js
CHANGED
|
@@ -1,6 +1,9 @@
|
|
|
1
|
+
import { GLOBAL_BOOLEAN_FLAGS_WITH_OPTIONAL_VALUES, VALUE_FLAGS } from "../../../argv-grammar.js";
|
|
2
|
+
import { isOpenNavigationCommand } from "../../../command-taxonomy.js";
|
|
1
3
|
import { getAgentBrowserProcessTimeoutMs } from "../../../process.js";
|
|
2
4
|
import { parseValidBatchStepEntries } from "../../batch-stdin.js";
|
|
3
|
-
const
|
|
5
|
+
const POSITIONAL_VALUE_FLAGS = new Set([...VALUE_FLAGS, "--llms"]);
|
|
6
|
+
const COMMAND_PROCESS_TIMEOUT_GRACE_MS = 5_000;
|
|
4
7
|
function parseMillisecondsToken(token) {
|
|
5
8
|
if (token === undefined || !/^\d+$/.test(token)) {
|
|
6
9
|
return undefined;
|
|
@@ -8,47 +11,98 @@ function parseMillisecondsToken(token) {
|
|
|
8
11
|
const parsed = Number(token);
|
|
9
12
|
return Number.isSafeInteger(parsed) ? parsed : undefined;
|
|
10
13
|
}
|
|
11
|
-
function
|
|
12
|
-
if (commandTokens[0] !== "wait")
|
|
14
|
+
function findCommandTimeoutMs(commandTokens) {
|
|
15
|
+
if (commandTokens[0] !== "wait" && commandTokens[0] !== "read")
|
|
13
16
|
return undefined;
|
|
14
|
-
}
|
|
15
17
|
for (let index = 1; index < commandTokens.length; index += 1) {
|
|
16
18
|
const token = commandTokens[index];
|
|
17
|
-
if (token === "--timeout")
|
|
19
|
+
if (token === "--timeout")
|
|
18
20
|
return parseMillisecondsToken(commandTokens[index + 1]);
|
|
19
|
-
|
|
20
|
-
if (token.startsWith("--timeout=")) {
|
|
21
|
+
if (token.startsWith("--timeout="))
|
|
21
22
|
return parseMillisecondsToken(token.slice("--timeout=".length));
|
|
22
|
-
}
|
|
23
23
|
}
|
|
24
|
-
const firstWaitArgument = commandTokens[1];
|
|
25
|
-
|
|
26
|
-
|
|
24
|
+
const firstWaitArgument = commandTokens[0] === "wait" ? commandTokens[1] : undefined;
|
|
25
|
+
return firstWaitArgument && !firstWaitArgument.startsWith("-") ? parseMillisecondsToken(firstWaitArgument) : undefined;
|
|
26
|
+
}
|
|
27
|
+
function findFirstPositionalArgument(commandTokens) {
|
|
28
|
+
for (let index = 1; index < commandTokens.length; index += 1) {
|
|
29
|
+
const token = commandTokens[index];
|
|
30
|
+
const flag = token.split("=", 1)[0];
|
|
31
|
+
if (POSITIONAL_VALUE_FLAGS.has(flag)) {
|
|
32
|
+
if (!token.includes("="))
|
|
33
|
+
index += 1;
|
|
34
|
+
continue;
|
|
35
|
+
}
|
|
36
|
+
if (GLOBAL_BOOLEAN_FLAGS_WITH_OPTIONAL_VALUES.has(flag) && ["true", "false"].includes(commandTokens[index + 1] ?? "")) {
|
|
37
|
+
index += 1;
|
|
38
|
+
continue;
|
|
39
|
+
}
|
|
40
|
+
if (!token.startsWith("-"))
|
|
41
|
+
return token;
|
|
27
42
|
}
|
|
28
43
|
return undefined;
|
|
29
44
|
}
|
|
30
|
-
function
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
45
|
+
function readUsesActivePageUrl(commandTokens) {
|
|
46
|
+
return findFirstPositionalArgument(commandTokens) === undefined && commandTokens.some((token) => token === "--require-md" || token === "--llms" || token.startsWith("--llms="));
|
|
47
|
+
}
|
|
48
|
+
function readRequestBudget(commandTokens, activePageUrl) {
|
|
49
|
+
const target = findFirstPositionalArgument(commandTokens) ?? activePageUrl;
|
|
50
|
+
if (target === undefined)
|
|
51
|
+
return 1;
|
|
52
|
+
let url;
|
|
53
|
+
try {
|
|
54
|
+
url = new URL(target.includes("://") ? target : `https://${target}`);
|
|
55
|
+
}
|
|
56
|
+
catch {
|
|
57
|
+
return 1;
|
|
34
58
|
}
|
|
35
|
-
|
|
59
|
+
const ancestorCount = url.pathname.split("/").filter(Boolean).length + 1;
|
|
60
|
+
if (commandTokens.some((token) => token === "--llms" || token.startsWith("--llms=")))
|
|
61
|
+
return ancestorCount;
|
|
62
|
+
if (commandTokens.includes("--raw"))
|
|
63
|
+
return 1;
|
|
64
|
+
return ancestorCount + 3;
|
|
65
|
+
}
|
|
66
|
+
function commandTimeoutBudgetMs(commandTokens, activePageUrl) {
|
|
67
|
+
const timeoutMs = findCommandTimeoutMs(commandTokens);
|
|
68
|
+
if (timeoutMs === undefined)
|
|
69
|
+
return undefined;
|
|
70
|
+
return commandTokens[0] === "read" ? timeoutMs * readRequestBudget(commandTokens, activePageUrl) : timeoutMs;
|
|
71
|
+
}
|
|
72
|
+
function findCommandTimeoutBudgetMs(commandTokens, stdin, activePageUrl) {
|
|
73
|
+
const directTimeout = commandTimeoutBudgetMs(commandTokens, activePageUrl);
|
|
74
|
+
if (directTimeout !== undefined)
|
|
75
|
+
return directTimeout;
|
|
76
|
+
if (commandTokens[0] !== "batch" || stdin === undefined)
|
|
36
77
|
return undefined;
|
|
78
|
+
let batchTimeoutTotal = 0;
|
|
79
|
+
let batchPageUrl = activePageUrl;
|
|
80
|
+
for (const { step } of parseValidBatchStepEntries(stdin)) {
|
|
81
|
+
batchTimeoutTotal += commandTimeoutBudgetMs(step, batchPageUrl) ?? 0;
|
|
82
|
+
if (isOpenNavigationCommand(step[0]))
|
|
83
|
+
batchPageUrl = findFirstPositionalArgument(step);
|
|
37
84
|
}
|
|
38
|
-
|
|
85
|
+
return batchTimeoutTotal === 0 ? undefined : batchTimeoutTotal;
|
|
86
|
+
}
|
|
87
|
+
export function commandTimeoutNeedsActivePageUrl(commandTokens, stdin) {
|
|
88
|
+
if (commandTokens[0] === "read")
|
|
89
|
+
return findCommandTimeoutMs(commandTokens) !== undefined && readUsesActivePageUrl(commandTokens);
|
|
90
|
+
if (commandTokens[0] !== "batch" || stdin === undefined)
|
|
91
|
+
return false;
|
|
92
|
+
let hasKnownPageUrl = false;
|
|
39
93
|
for (const { step } of parseValidBatchStepEntries(stdin)) {
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
94
|
+
if (isOpenNavigationCommand(step[0]) && findFirstPositionalArgument(step))
|
|
95
|
+
hasKnownPageUrl = true;
|
|
96
|
+
if (!hasKnownPageUrl && findCommandTimeoutMs(step) !== undefined && step[0] === "read" && readUsesActivePageUrl(step))
|
|
97
|
+
return true;
|
|
44
98
|
}
|
|
45
|
-
return
|
|
99
|
+
return false;
|
|
46
100
|
}
|
|
47
|
-
export function
|
|
48
|
-
const
|
|
49
|
-
if (
|
|
101
|
+
export function getCommandAwareProcessTimeoutMs(commandTokens, stdin, activePageUrl) {
|
|
102
|
+
const timeoutBudgetMs = findCommandTimeoutBudgetMs(commandTokens, stdin, activePageUrl);
|
|
103
|
+
if (timeoutBudgetMs === undefined)
|
|
50
104
|
return undefined;
|
|
51
|
-
const neededTimeoutMs =
|
|
105
|
+
const neededTimeoutMs = timeoutBudgetMs + COMMAND_PROCESS_TIMEOUT_GRACE_MS;
|
|
52
106
|
const defaultProcessTimeoutMs = getAgentBrowserProcessTimeoutMs();
|
|
53
107
|
return neededTimeoutMs > defaultProcessTimeoutMs ? neededTimeoutMs : undefined;
|
|
54
108
|
}
|
|
@@ -7,7 +7,7 @@ import { tryDirectAnchorDownload } from "./prepare/direct-anchor-download.js";
|
|
|
7
7
|
import { tryNetworkRequestsPageFilter } from "./prepare/network-page-filter.js";
|
|
8
8
|
import { tryContainerScroll, tryPageScrollTo } from "./prepare/scroll-shims.js";
|
|
9
9
|
import { trySnapshotFilter } from "./prepare/snapshot-filter.js";
|
|
10
|
-
import {
|
|
10
|
+
import { commandTimeoutNeedsActivePageUrl, getCommandAwareProcessTimeoutMs } from "./prepare/wait-timeouts.js";
|
|
11
11
|
import { getPersistentSessionArtifactStore } from "./session-artifacts.js";
|
|
12
12
|
import { buildAgentBrowserResultCategoryDetails } from "../../results.js";
|
|
13
13
|
import { applyNamespaceToNextActions } from "../../results/next-actions.js";
|
|
@@ -15,7 +15,7 @@ import { buildSessionAwareStaleRefNextActions, buildSessionTabRecoveryNextAction
|
|
|
15
15
|
import { resolveVisibleRefActionFromSnapshot } from "../../results/selector-recovery.js";
|
|
16
16
|
import { extractRefSnapshotFromData } from "../../session-page-state.js";
|
|
17
17
|
import { buildExecutionPlan, createFreshSessionName, extractCommandTokens, redactInvocationArgs, } from "../../runtime.js";
|
|
18
|
-
import { applyOpenResultTabCorrection, buildManagedSessionOutcome, buildPinnedBatchPlan, buildSessionDetailFields, buildStaleRefPreflight, getSessionContextKey, collectAnySessionTabSelection, collectSessionTabSelection, getGuardedRefUsage, getTraceOwnerGuardMessage, runSessionCommandData, shouldPinSessionTabForCommand, } from "./session-state.js";
|
|
18
|
+
import { applyOpenResultTabCorrection, buildManagedSessionOutcome, buildPinnedBatchPlan, buildSessionDetailFields, buildStaleRefPreflight, getSessionContextKey, extractStringResultField, collectAnySessionTabSelection, collectSessionTabSelection, getGuardedRefUsage, getTraceOwnerGuardMessage, runSessionCommandData, shouldPinSessionTabForCommand, } from "./session-state.js";
|
|
19
19
|
import { parseBatchStdinJsonArray } from "../batch-stdin.js";
|
|
20
20
|
import { buildElectronHostFailureResult, getElectronLaunchFailureCategory, redactRecoveryHint } from "./final-result.js";
|
|
21
21
|
import { prepareClickDispatchProbe } from "./click-dispatch.js";
|
|
@@ -296,7 +296,7 @@ export async function resolveSemanticActionVisibleRefArgs(options) {
|
|
|
296
296
|
return resolveSemanticActionVisibleRefArgsFromSnapshot(options.compiled, snapshotData);
|
|
297
297
|
}
|
|
298
298
|
export async function prepareBrowserRun(options) {
|
|
299
|
-
const { cwd,
|
|
299
|
+
const { cwd, onUpdate, params, signal, state } = options;
|
|
300
300
|
const { sessionPageState, traceOwners, managedSessionBaseName, ephemeralSessionSeed } = state;
|
|
301
301
|
let freshSessionOrdinal = state.freshSessionOrdinal;
|
|
302
302
|
const { compiledElectron, compiledJob, compiledNetworkSourceLookup, compiledQaPreset, compiledSemanticAction, compiledSourceLookup, redactedArgs, redactedCompiledElectron, redactedCompiledJob, redactedCompiledNetworkSourceLookup, redactedCompiledQaPreset, redactedCompiledSemanticAction, redactedCompiledSourceLookup, toolArgs, toolStdin, } = normalizeRunInput(options.input);
|
|
@@ -725,7 +725,15 @@ export async function prepareBrowserRun(options) {
|
|
|
725
725
|
const clickDispatchProbe = pinnedBatchUnwrapMode === undefined && compiledElectron === undefined
|
|
726
726
|
? await prepareClickDispatchProbe({ commandTokens, cwd, namespace: executionPlan.namespace, refSnapshot: promptRefSnapshot, sessionName: executionPlan.sessionName, signal })
|
|
727
727
|
: undefined;
|
|
728
|
-
|
|
728
|
+
let readTimeoutPageUrl = priorSessionTabTarget?.url;
|
|
729
|
+
if (options.params.timeoutMs === undefined && readTimeoutPageUrl === undefined && executionPlan.sessionName && commandTimeoutNeedsActivePageUrl(commandTokens, processStdin)) {
|
|
730
|
+
try {
|
|
731
|
+
const data = await runSessionCommandData({ args: ["get", "url"], cwd, namespace: executionPlan.namespace, sessionName: executionPlan.sessionName, signal });
|
|
732
|
+
readTimeoutPageUrl = extractStringResultField(data, "result") ?? extractStringResultField(data, "url");
|
|
733
|
+
}
|
|
734
|
+
catch { }
|
|
735
|
+
}
|
|
736
|
+
const processTimeoutMs = options.params.timeoutMs ?? getDialogAwareProcessTimeoutMs(commandTokens, promptRefSnapshot, processStdin) ?? getCommandAwareProcessTimeoutMs(commandTokens, processStdin, readTimeoutPageUrl);
|
|
729
737
|
const redactedProcessArgs = redactInvocationArgs(processArgs);
|
|
730
738
|
const scrollAmount = Number(commandTokens.find((token) => /^\d+(?:\.\d+)?$/.test(token)));
|
|
731
739
|
const shouldProbeScrollNoop = executionPlan.commandInfo.command === "scroll" && executionPlan.startupScopedFlags.length === 0 && (state.managedSessionActive || sessionMode === "fresh") && (!Number.isFinite(scrollAmount) || scrollAmount >= 500);
|
|
@@ -20,7 +20,7 @@ export const QUICK_START_GUIDELINES = [
|
|
|
20
20
|
"Locator-first clicks/fills and native select changes without hand-building argv: { semanticAction: { action: \"click\", locator: \"text\", value: \"Close\" } }, { semanticAction: { action: \"fill\", locator: \"label\", value: \"Email\", text: \"user@example.com\" } }, direct current targets such as { semanticAction: { action: \"fill\", selector: \"@e1\", text: \"prompt\" } }, or { semanticAction: { action: \"select\", selector: \"#flavor\", value: \"chocolate\" } }; add semanticAction.session when targeting a named upstream browser session; details.compiledSemanticAction shows the semantic target, while details.effectiveArgs may show a resolved current @ref for active-session role/name click/check/fill actions to avoid hidden duplicate matches; semanticAction does not expose uncheck while upstream find ... uncheck is not runtime-supported, so use raw uncheck with a stable selector or current ref; selector-not-found failures may append bounded click try-*-candidate next actions or, for fill misses with current editable refs, details.richInputRecovery with focus/click actions that do not copy fill text; stale-ref failures can return retry-semantic-action-after-stale-ref for compiled find actions when retry safety is provable.",
|
|
21
21
|
`Common advanced calls: { args: ["batch"], stdin: "[[\"open\",\"https://example.com\"],[\"snapshot\",\"-i\"]]" }, { job: { steps: [{ action: "open", url: "https://example.com" }, { action: "assertText", text: "Example Domain" }, { action: "screenshot", path: ".dogfood/example.png" }] } }, { qa: { url: "https://example.com", expectedText: "Example Domain", screenshotPath: ".dogfood/qa-example.png" } } (example.com smoke only; elsewhere match exact visible text from snapshot -i), { electron: { action: "list", query: "code" } }, { electron: { action: "launch", appName: "Visual Studio Code", handoff: "snapshot" } }, { electron: { action: "probe" } }, { qa: { attached: true, expectedText: "Explorer" } }, { args: ["eval", "--stdin"], stdin: "document.title", outputPath: "logs/page-title.json" }, { args: ["auth", "save", "name", "--password-stdin"], stdin: "<password from user-approved secret source>" }, { args: ["--profile", "Default", "open", "https://example.com/account"], sessionMode: "fresh" }, and { args: ["open", "--enable", "react-devtools", "https://example.com"], sessionMode: "fresh" }. For app pages with a native dropdown, job steps can include { action: "select", selector: "#flavor", value: "chocolate" } before the dependent assertion; for locator-friendly pages, job click/fill steps can use semantic locator fields such as { action: "fill", locator: "role", role: "searchbox", name: "Search", text: "agent browser" }; for human-paced input, job type steps can use { action: "type", selector: "#prompt", text: "hello", delayMs: 20, press: "Enter" }; delayed typing is capped at 200 characters per step, and generated per-character rows are compacted in visible batch prose while full rows remain in details.batchSteps.`,
|
|
22
22
|
"Constrained job navigation is explicit only: click (and select/submit flows that may navigate) does not prove the next page loaded; add assertUrl and/or assertText after navigation-prone steps before screenshot or later interactions. Keep jobs short around navigation, click, and rerender boundaries on dynamic React/product apps; avoid a whole checkout in one job. If a long job times out and details.timeoutPartialProgress shows a mutating incomplete step, inspect current page state and continue with a shorter job or single action instead of blindly retrying the mutating step. Example: { job: { steps: [{ action: \"open\", url: \"https://shop.example/checkout\" }, { action: \"fill\", selector: \"#email\", text: \"user@example.com\" }, { action: \"click\", selector: \"#continue\" }, { action: \"assertUrl\", url: \"**/shipping\" }, { action: \"assertText\", text: \"Shipping address\" }, { action: \"screenshot\", path: \".dogfood/shipping.png\" }] } }. Top-level click may add pageChangeSummary hints, but job never auto-inserts post-click asserts.",
|
|
23
|
-
"High-value command reference: click <selector> --new-tab opens link-like targets in a new tab; select <selector> <value...> changes native dropdown values; scroll <dir> [px] --selector <sel>, wrapper-handled scroll <selector> <dir> [px|percent] targets nested scrollers, and wrapper-handled scroll to end/top targets document scrolling; download <selector> <path> saves a file triggered by a click; get title/url need no selector; get text/html/value/count <selector> and get attr <selector> <name> read elements/page state (use body for whole-page text/html); screenshot [selector] [path] captures a page or element image; pdf <path> saves a PDF; tab list and tab <tab-id-or-label> inspect or recover the active tab; react tree, react inspect <fiberId>, react renders start/stop, and react suspense introspect React after --enable react-devtools; vitals [url] measures Core Web Vitals; pushstate <url> performs SPA navigation; tap <selector> and swipe <direction> [distance] support iOS/provider touch flows.",
|
|
23
|
+
"High-value command reference: click <selector> --new-tab opens link-like targets in a new tab; select <selector> <value...> changes native dropdown values; scroll <dir> [px] --selector <sel>, wrapper-handled scroll <selector> <dir> [px|percent] targets nested scrollers, and wrapper-handled scroll to end/top targets document scrolling; download <selector> <path> saves a file triggered by a click; read [url] returns agent-readable text (explicit URLs prefer markdown without launching Chrome; omit the URL for rendered active-tab DOM); get title/url need no selector; get text/html/value/count <selector> and get attr <selector> <name> read elements/page state (use body for whole-page text/html); screenshot [selector] [path] captures a page or element image; pdf <path> saves a PDF; tab list and tab <tab-id-or-label> inspect or recover the active tab; react tree, react inspect <fiberId>, react renders start/stop, and react suspense introspect React after --enable react-devtools; vitals [url] measures Core Web Vitals; pushstate <url> performs SPA navigation; tap <selector> and swipe <direction> [distance] support iOS/provider touch flows.",
|
|
24
24
|
"For artifact-producing commands, read the visible artifact block and details.artifactVerification before using files: check requested path, absolute path, existence, size bytes, artifact kind, optional mediaType, status, optional limitation, and verified/missing/pending/unverified counts. details.artifacts contains per-file metadata; record start rows are pending/openRecording until record stop writes the target. The wrapper creates parent directories for direct artifact paths and can save simple loopback HTTP(S) anchor downloads directly to the requested path before upstream download fallback. Browser close does not delete explicit saved files; if close reports details.artifactCleanup, use host file tools to remove paths listed in explicitArtifactPaths (when non-empty) after inspection. If close fails with details.promptGuard.reason=requested-artifacts-missing-before-close, save the exact required artifact path before closing. For annotated screenshots inside batch, put --annotate in top-level args (for example { args: [\"--annotate\", \"batch\"], stdin: \"[[\\\"screenshot\\\",\\\"/tmp/page.png\\\"]]\" }) rather than inside the screenshot step; if annotation labels crowd a dense page, use a scoped or non-annotated screenshot plus snapshot refs instead.",
|
|
25
25
|
"When details.nextActions is present, prefer those exact native agent_browser follow-up payloads over prose guidance; they may include args, stdin, sessionMode, networkSourceLookup, safety notes, or artifactPath for saved files.",
|
|
26
26
|
];
|
|
@@ -50,7 +50,7 @@ export const SHARED_BROWSER_PLAYBOOK_GUIDELINES = [
|
|
|
50
50
|
"For dialogs and frames, use dialog status/accept/dismiss and frame <selector|main> through native args; dialog commands and eval snippets that look like alert/confirm/prompt/dialog triggers are shorter-bounded than normal browser calls, and timed-out dialog-like interactions may add inspect-dialog-after-timeout, dismiss-dialog-after-timeout, or recover-fresh-session-after-dialog-timeout nextActions. When --confirm-actions produces a pending confirmation, use details.nextActions or exact confirm <id> / deny <id> calls instead of inventing ids.",
|
|
51
51
|
"If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or qa.attached; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait <selector> --state hidden. Use electron.probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort: use explicit --timeout or top-level timeoutMs for legitimately slow waits, and treat a successful payload like \"waited\":\"timeout\" as elapsed time only—verify completion with an observed condition, fresh snapshot, or screenshot.",
|
|
52
52
|
"For feed, timeline, or inbox reading tasks, focus on the main timeline/list region and read the first item there rather than unrelated composer or sidebar content.",
|
|
53
|
-
"For read-only browsing tasks,
|
|
53
|
+
"For read-only browsing tasks, use read <url> for documentation or other unstructured text without launching Chrome, or read with no URL for rendered active-tab DOM. Prefer the current snapshot, structured ref labels, getters, or scoped eval --stdin when you need interactive structure or targeted page state. Only click into media viewers, detail routes, or new pages when the current view does not contain the needed information.",
|
|
54
54
|
"For downloads, prefer download <selector> <path> when an element click should save a file; simple loopback anchor downloads are saved to the requested path when the wrapper can resolve an HTTP(S) href. Do not rely on click alone when you need the downloaded file on disk.",
|
|
55
55
|
"On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or semanticAction/job select) instead of clicking option refs; for custom comboboxes, a click/semanticAction may only focus the field, so re-snapshot and fall back to type, press Enter/arrow keys, or visible option refs.",
|
|
56
56
|
"When using eval --stdin, scope checks and actions to the target element or route whenever possible instead of relying on broad page-wide text heuristics.",
|
|
@@ -79,13 +79,6 @@ export const WRAPPER_TAB_RECOVERY_BEHAVIOR = [
|
|
|
79
79
|
"For sessions with observed tab-drift risk, after a successful command on a known target tab, agent_browser also best-effort restores that intended tab if a restored/background tab steals focus after the command completes. Routine same-session commands skip this post-command tab-list probe.",
|
|
80
80
|
"If a known session target unexpectedly reports about:blank, agent_browser best-effort re-selects the prior intended target when it still exists; if recovery fails, it records the observed about:blank target and reports exact recovery guidance instead of treating the prior page as active.",
|
|
81
81
|
];
|
|
82
|
-
export function buildSharedBrowserPlaybookGuidelines(options) {
|
|
83
|
-
return [
|
|
84
|
-
SHARED_BROWSER_PLAYBOOK_GUIDELINES[0],
|
|
85
|
-
...(options.includeWebSearch ? [WEB_SEARCH_PROMPT_GUIDELINE] : []),
|
|
86
|
-
...SHARED_BROWSER_PLAYBOOK_GUIDELINES.slice(1),
|
|
87
|
-
];
|
|
88
|
-
}
|
|
89
82
|
/** Tier A: always-on tool promptGuidelines (keep small; Tier B lives in SHARED_BROWSER_PLAYBOOK_GUIDELINES and docs). */
|
|
90
83
|
export const RUNTIME_PROMPT_GUIDELINES = [
|
|
91
84
|
"Use agent_browser with one input mode: args, semanticAction, job, qa, sourceLookup/networkSourceLookup, or electron. stdin only for batch/eval/auth/wrapper batch; electron rejects stdin; never pass --json.",
|
|
@@ -93,7 +86,7 @@ export const RUNTIME_PROMPT_GUIDELINES = [
|
|
|
93
86
|
"Use agent_browser sessionMode=fresh for launch-scoped flags, including --allowed-domains; never put --session-mode in args. Use requested/configured profiles only; on profile failures run profiles/doctor. Profile content is model-visible.",
|
|
94
87
|
"For agent_browser artifacts, use exact user paths and verify details.artifactVerification/details.artifacts before claiming success. Save details.promptGuard-required artifacts before close; record stop needs ffmpeg; close keeps files; waited:timeout is not proof.",
|
|
95
88
|
"When agent_browser details.nextActions exists, use exact payloads over guessed selectors/prose. Dense snapshots: check Omitted high-value controls/highValueControlRefIds. Dashboards: verify scroll with screenshot/snapshot.",
|
|
96
|
-
"For agent_browser extraction
|
|
89
|
+
"For agent_browser extraction: read <url> for docs/text; read for active-tab DOM; get title/url; get text/html/value/count <selector>; get attr <selector> <name>; eval --stdin for targeted state. Batch 3+ getters; heed visibility warnings.",
|
|
97
90
|
];
|
|
98
91
|
export function buildBrowserExecutablePathGuideline(executablePath) {
|
|
99
92
|
if (!executablePath)
|
|
@@ -84,6 +84,10 @@ const COMMAND_PRESENTERS = {
|
|
|
84
84
|
summary: (_commandInfo, data) => Array.isArray(data) ? `Chrome profiles: ${data.length}` : undefined,
|
|
85
85
|
text: (_commandInfo, data) => Array.isArray(data) ? formatProfilesText(data, "Chrome profiles") : undefined,
|
|
86
86
|
},
|
|
87
|
+
read: {
|
|
88
|
+
summary: (_commandInfo, data) => isRecord(data) && typeof (data.finalUrl ?? data.url) === "string" ? `Read: ${data.finalUrl ?? data.url}` : undefined,
|
|
89
|
+
text: (_commandInfo, data) => isRecord(data) && typeof data.content === "string" ? redactModelFacingText(data.content) : undefined,
|
|
90
|
+
},
|
|
87
91
|
screenshot: {
|
|
88
92
|
summary: (_commandInfo, data) => isRecord(data) && typeof data.path === "string" ? `Screenshot saved: ${data.path}` : undefined,
|
|
89
93
|
text: (_commandInfo, data) => isRecord(data) ? getScreenshotSummary(data) : undefined,
|
|
@@ -779,7 +779,6 @@ export function buildExecutionPlan(args, options) {
|
|
|
779
779
|
const startupScopedFlags = getStartupScopedFlags(args).filter((flag) => !(flag === "--namespace" && explicitNamespace === options.managedSessionNamespace));
|
|
780
780
|
const plainTextInspection = isPlainTextInspectionArgs(args);
|
|
781
781
|
const argvDescriptor = parseArgvDescriptor(args);
|
|
782
|
-
const commandTokens = argvDescriptor.commandTokens;
|
|
783
782
|
const commandInfo = argvDescriptor.commandInfo;
|
|
784
783
|
const commandNeedsManagedSession = !plainTextInspection && needsManagedSession(argvDescriptor);
|
|
785
784
|
const effectiveArgs = plainTextInspection ? [...args] : args.includes("--json") ? [] : ["--json"];
|
package/docs/ARCHITECTURE.md
CHANGED
|
@@ -109,7 +109,7 @@ Why:
|
|
|
109
109
|
- keeps reload and exact-session relaunch validation tied to Pi's configured-source lifecycle instead of an isolated quick-test path, while `session_tree` state changes stay covered by focused extension harness tests
|
|
110
110
|
- keeps the published tarball focused on the package manifest, extension code, canonical docs, and license
|
|
111
111
|
|
|
112
|
-
The published package should exclude agent-only and
|
|
112
|
+
The published package should exclude agent-only and internal planning materials such as `AGENTS.md` and `docs/plans/`.
|
|
113
113
|
|
|
114
114
|
## Session model
|
|
115
115
|
|
|
@@ -206,7 +206,7 @@ Upstream restore-state persistence remains upstream-owned. The wrapper passes `A
|
|
|
206
206
|
- subprocess execution and JSON parsing through `buildAgentBrowserProcessEnv` in `extensions/agent-browser/lib/process.ts`: copies the parent process environment so user-approved provider credentials and other runtime variables reach upstream, then applies wrapper overrides such as the managed socket directory and clamped default operation timeout
|
|
207
207
|
- clear missing-binary errors
|
|
208
208
|
- compact result summaries, including presentation-time redaction: stateful browser-context commands (`auth`, `cookies`, `storage`, `dialog`, `frame`, `state`) use field-aware value redaction and compact formatters, while other structured upstream JSON (for example `network`, `diff`, `trace` / `profiler` / `record`, `console` / `errors` / `highlight` / `inspect` / `clipboard`, `stream`, `dashboard`, and `chat`) is passed through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation.ts` so model-facing `details.data` and batch roll-ups stay compact and do not echo bearer tokens, proxy passwords, or similar fields verbatim; `redactInvocationArgs` in `extensions/agent-browser/lib/runtime.ts` masks trailing values for sensitive global flags such as `--body`, `--headers`, `--password`, and `--proxy`, preserves positional rules for `cookies set` and `storage local|session set`, and nested `batch` steps use the same argv and error-body scrubbing before echoing commands or errors
|
|
209
|
-
- bounded machine-readable outcome metadata on tool `details` (`resultCategory`, `successCategory`, `failureCategory`, optional `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`, optional `artifactVerification` with the same shape on each successful `batchSteps[]` row) so agents can branch without parsing prose; enums, classifier precedence, and generic follow-up payloads are implemented under `extensions/agent-browser/lib/results/` in focused modules (`contracts.ts` for shared types, `categories.ts` for `classifyAgentBrowserSuccessCategory` / `classifyAgentBrowserFailureCategory` / `buildAgentBrowserResultCategoryDetails`, `action-recommendations.ts` for `buildAgentBrowserNextActions`, `next-actions.ts` for the `AgentBrowserNextAction` shape and merge helpers, `recovery-actions.ts` for recovery id registries and `buildRecoveryNextActions`, `network.ts` for `classifyNetworkRequestFailure` / `summarizeNetworkFailures`, and related helpers
|
|
209
|
+
- bounded machine-readable outcome metadata on tool `details` (`resultCategory`, `successCategory`, `failureCategory`, optional `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`, optional `artifactVerification` with the same shape on each successful `batchSteps[]` row) so agents can branch without parsing prose; enums, classifier precedence, and generic follow-up payloads are implemented under `extensions/agent-browser/lib/results/` in focused modules (`contracts.ts` for shared types, `categories.ts` for `classifyAgentBrowserSuccessCategory` / `classifyAgentBrowserFailureCategory` / `buildAgentBrowserResultCategoryDetails`, `action-recommendations.ts` for `buildAgentBrowserNextActions`, `next-actions.ts` for the `AgentBrowserNextAction` shape and merge helpers, `recovery-actions.ts` for recovery id registries and `buildRecoveryNextActions`, `network.ts` for `classifyNetworkRequestFailure` / `summarizeNetworkFailures`, and related helpers). Per-session tab target, `refSnapshot` alignment, invalidation, and tab pinning observations flow through `extensions/agent-browser/lib/session-page-state.ts` from `extensions/agent-browser/index.ts`. Compact page-change summaries and artifact verification rollups are built in `extensions/agent-browser/lib/results/presentation.ts` (`buildPageChangeSummary`, `buildArtifactVerificationSummary`), and the human contract lives in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details). Real Pi custom tools otherwise only mark a row failed when `execute` throws, so the extension also registers `pi.on("tool_result", …)` and patches `agent_browser` results whose `details.resultCategory` is `failure` to set `isError: true`. Prose results also receive a short category notice, while caller-requested `--json` results with parseable JSON content keep that text unchanged so JSONL transcripts, UI affordances, and the machine-readable contract stay aligned for wrapper-side reclassifications such as `qa-failure` (`buildAgentBrowserToolResultPatch` in `extensions/agent-browser/lib/pi-tool-rendering.ts`; transcript semantics in the same contract doc)
|
|
210
210
|
- inline screenshots/images for the plain `screenshot` command; other image-like saves (for example `diff screenshot`) still appear in `details.artifacts` and summaries but are not auto-inlined as Pi image attachments (see [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details))
|
|
211
211
|
- lightweight session convenience
|
|
212
212
|
- docs, including a repo-readable command reference that mirrors the blocked direct-binary help path closely enough for normal agent work
|
|
@@ -18,13 +18,23 @@ This project intentionally blocks normal `agent-browser` bash usage in most agen
|
|
|
18
18
|
|
|
19
19
|
<!-- agent-browser-capability-baseline:start upstream-baseline -->
|
|
20
20
|
<!-- Generated from scripts/agent-browser-capability-baseline.mjs. Run `npm run docs -- command-reference write` to update. Do not edit manually. -->
|
|
21
|
-
This reference is baselined to the locally installed `agent-browser 0.32.
|
|
21
|
+
This reference is baselined to the locally installed `agent-browser 0.32.2` command/help surface, audited against vercel-labs/agent-browser@6ede7a9470ac4b681cabf838af8668b9aa99e957. Upstream `agent-browser` remains the source of truth for command semantics; this file is the local fallback for Pi agent sessions where direct binary help is blocked or discouraged.
|
|
22
22
|
|
|
23
23
|
The lightweight drift check is `npm run verify -- command-reference`. Run it whenever the installed upstream `agent-browser` version changes or this reference is edited.
|
|
24
24
|
|
|
25
25
|
Use `npm run benchmark:agent-browser` or `npm run verify -- benchmark` before and after agent-facing workflow abstractions to measure task success, tool calls, model-visible output size, stale-ref behavior, artifact success, failure-category coverage, and elapsed-time estimates.
|
|
26
26
|
<!-- agent-browser-capability-baseline:end upstream-baseline -->
|
|
27
27
|
|
|
28
|
+
### Upstream 0.32.2 rebaseline
|
|
29
|
+
|
|
30
|
+
The 0.32.1 and 0.32.2 releases only update the separate `@agent-browser/eve` integration; the CLI/help/schema surface is unchanged:
|
|
31
|
+
|
|
32
|
+
- 0.32.1 standardized lowercase eve branding and widened compatibility beyond the original pinned peer range.
|
|
33
|
+
- 0.32.2 targets eve 0.25.1+, adopts eve's source/dist extension manifest, updates its example to stable AI SDK packages, and aligns tests with eve's scoped config registry.
|
|
34
|
+
- This Pi extension still does not bundle `@agent-browser/eve` or add an eve-specific input mode.
|
|
35
|
+
|
|
36
|
+
The current audit also closes a command-reference/presentation gap for upstream `read [url]`, which has existed since 0.30.0: the capability baseline now samples `read --help`, native results render `data.content` instead of only the fetched URL, explicit fetch metadata cannot replace the active browser tab target, and `read --timeout <ms>` extends the wrapper subprocess budget across upstream's per-request `.md` and ancestor-`llms.txt` fallback sequence.
|
|
37
|
+
|
|
28
38
|
### Upstream 0.32.0 rebaseline
|
|
29
39
|
|
|
30
40
|
The 0.32.0 rebaseline hardens domain containment and fixes completed-page waits without adding a new native Pi input mode:
|
|
@@ -247,12 +257,16 @@ Successful `snapshot -i` results can also surface `Possible overlay blockers` wh
|
|
|
247
257
|
### Extract page data
|
|
248
258
|
|
|
249
259
|
```json
|
|
260
|
+
{ "args": ["read", "https://example.com/docs", "--filter", "authentication"] }
|
|
261
|
+
{ "args": ["read"] }
|
|
250
262
|
{ "args": ["get", "title"] }
|
|
251
263
|
{ "args": ["get", "url"] }
|
|
252
264
|
{ "args": ["get", "text", "main"] }
|
|
253
265
|
{ "args": ["eval", "--stdin"], "stdin": "document.title" }
|
|
254
266
|
```
|
|
255
267
|
|
|
268
|
+
Use `read [url]` for documentation and other unstructured text. `read <url> --raw` preserves the response body, `read <url> --require-md` requires `text/markdown`, `read <url> --llms <index|full>` reads the nearest ancestor llms index/full file, `read <url> --outline` emits headings, `read <url> --filter <text>` narrows matching sections/headings/links, and `read <url> --timeout <ms>` changes the request timeout. Explicit URL reads prefer markdown, try a `.md` path and nearby `llms.txt` links, then fall back to readable HTML without launching Chrome. Omit the URL to read rendered active-tab DOM, including current browser auth and client-side state; `--llms` / `--require-md` without a URL instead fetch from the active tab URL. The wrapper renders `data.content` first, retains source/content-type/status/final-URL metadata in `details.data`, keeps fetched URLs from replacing the active browser tab target, and extends its subprocess watchdog for explicit long read timeouts.
|
|
269
|
+
|
|
256
270
|
When you already know several visible refs or selectors, extract them in one `batch` call instead of many serial getter calls:
|
|
257
271
|
|
|
258
272
|
```json
|
|
@@ -561,6 +575,7 @@ Skill-source debugging note: upstream honors `AGENT_BROWSER_SKILLS_DIR` as an ov
|
|
|
561
575
|
| --- | --- |
|
|
562
576
|
| `open [url]` | Launch the browser and optionally navigate. URL-less `open` stays on `about:blank` so agents can stage routes, cookies, or init scripts before first navigation. |
|
|
563
577
|
| `open <url>` | Navigate to a URL; `goto <url>` and `navigate <url>` are equivalent navigation aliases when a URL is present. |
|
|
578
|
+
| `read [url]` | Fetch agent-readable text from an explicit URL without launching Chrome, or omit the URL to read rendered active-tab DOM. Supports `--raw`, `--require-md`, `--llms <index|full>`, `--outline`, `--filter <text>`, and `--timeout <ms>`. |
|
|
564
579
|
| `click <sel>` | Click an element or `@ref`. |
|
|
565
580
|
| `click <sel> --new-tab` | Click a link/control while requesting a new tab. |
|
|
566
581
|
| `dblclick <sel>` | Double-click an element. |
|
|
@@ -934,14 +949,14 @@ Other useful environment variables include `AGENT_BROWSER_DEFAULT_TIMEOUT`, `AGE
|
|
|
934
949
|
<!-- agent-browser-capability-baseline:start capability-token-baseline -->
|
|
935
950
|
<!-- Generated from scripts/agent-browser-capability-baseline.mjs. Run `npm run docs -- command-reference write` to update. Do not edit manually. -->
|
|
936
951
|
<details>
|
|
937
|
-
<summary>Generated verifier capability baseline for agent-browser 0.32.
|
|
952
|
+
<summary>Generated verifier capability baseline for agent-browser 0.32.2</summary>
|
|
938
953
|
|
|
939
954
|
This generated block is review data for maintainers. The human-authored reference sections above remain the readable command guide.
|
|
940
955
|
|
|
941
956
|
#### Source evidence
|
|
942
957
|
- repository: `vercel-labs/agent-browser`
|
|
943
|
-
- upstream HEAD: `
|
|
944
|
-
- upstream package version: `0.32.
|
|
958
|
+
- upstream HEAD: `6ede7a9470ac4b681cabf838af8668b9aa99e957`
|
|
959
|
+
- upstream package version: `0.32.2`
|
|
945
960
|
- inspected: `agent-browser --version`
|
|
946
961
|
- inspected: `agent-browser --help`
|
|
947
962
|
- inspected: `selected agent-browser <command> --help output`
|
|
@@ -952,11 +967,14 @@ This generated block is review data for maintainers. The human-authored referenc
|
|
|
952
967
|
- inspected: `agent-browser.schema.json`
|
|
953
968
|
- inspected: `cli/src/commands.rs`
|
|
954
969
|
- inspected: `cli/src/flags.rs`
|
|
970
|
+
- inspected: `cli/src/read.rs`
|
|
955
971
|
- inspected: `cli/src/doctor/webgpu.rs`
|
|
956
972
|
- inspected: `cli/src/native/actions.rs`
|
|
957
973
|
- inspected: `cli/src/native/daemon.rs`
|
|
958
974
|
- inspected: `docs/src/app/webgpu/page.mdx`
|
|
959
975
|
- inspected: `packages/@agent-browser/eve/README.md`
|
|
976
|
+
- inspected: `packages/@agent-browser/eve/package.json`
|
|
977
|
+
- inspected: `packages/@agent-browser/eve/test/extension.test.mjs`
|
|
960
978
|
- inspected: `packages/@agent-browser/sandbox/README.md`
|
|
961
979
|
- inspected: `packages/@agent-browser/sandbox/src/shared.ts`
|
|
962
980
|
- inspected: `packages/@agent-browser/sandbox/src/vercel.ts`
|
|
@@ -969,6 +987,7 @@ This generated block is review data for maintainers. The human-authored referenc
|
|
|
969
987
|
- core skill full: `agent-browser skills get core --full`
|
|
970
988
|
- vercel sandbox skill full: `agent-browser skills get vercel-sandbox --full`
|
|
971
989
|
- open help: `agent-browser open --help`
|
|
990
|
+
- read help: `agent-browser read --help`
|
|
972
991
|
- click help: `agent-browser click --help`
|
|
973
992
|
- key help: `agent-browser key --help`
|
|
974
993
|
- scroll help: `agent-browser scroll --help`
|
|
@@ -1020,7 +1039,7 @@ This generated block is review data for maintainers. The human-authored referenc
|
|
|
1020
1039
|
|
|
1021
1040
|
#### Inventory sections
|
|
1022
1041
|
- Built-in skills: 15 human-doc token(s), 15 upstream token(s)
|
|
1023
|
-
- Core page, element, navigation, and extraction commands:
|
|
1042
|
+
- Core page, element, navigation, and extraction commands: 81 human-doc token(s), 82 upstream token(s)
|
|
1024
1043
|
- Sessions, state, tabs, frames, dialogs, and windows: 24 human-doc token(s), 20 upstream token(s)
|
|
1025
1044
|
- Network, storage, artifacts, diagnostics, and performance: 43 human-doc token(s), 53 upstream token(s)
|
|
1026
1045
|
- Batch, auth, confirmations, setup, dashboard, devices, and AI commands: 33 human-doc token(s), 37 upstream token(s)
|
|
@@ -1049,6 +1068,13 @@ This generated block is review data for maintainers. The human-authored referenc
|
|
|
1049
1068
|
- `open <url>`
|
|
1050
1069
|
- `goto <url>`
|
|
1051
1070
|
- `navigate <url>`
|
|
1071
|
+
- `read [url]`
|
|
1072
|
+
- `read <url> --raw`
|
|
1073
|
+
- `read <url> --require-md`
|
|
1074
|
+
- `read <url> --llms <index|full>`
|
|
1075
|
+
- `read <url> --outline`
|
|
1076
|
+
- `read <url> --filter <text>`
|
|
1077
|
+
- `read <url> --timeout <ms>`
|
|
1052
1078
|
- `click <sel>`
|
|
1053
1079
|
- `click <sel> --new-tab`
|
|
1054
1080
|
- `dblclick <sel>`
|
|
@@ -1388,6 +1414,14 @@ This generated block is review data for maintainers. The human-authored referenc
|
|
|
1388
1414
|
- open help: `open [url]`
|
|
1389
1415
|
- open help: `aliases still require a URL.`
|
|
1390
1416
|
- root help: `open <url>`
|
|
1417
|
+
- root help: `read [url]`
|
|
1418
|
+
- read help: `read [url]`
|
|
1419
|
+
- read help: `--raw`
|
|
1420
|
+
- read help: `--require-md`
|
|
1421
|
+
- read help: `--llms <index|full>`
|
|
1422
|
+
- read help: `--outline`
|
|
1423
|
+
- read help: `--filter <text>`
|
|
1424
|
+
- read help: `--timeout <ms>`
|
|
1391
1425
|
- root help: `click <sel>`
|
|
1392
1426
|
- click help: `--new-tab`
|
|
1393
1427
|
- root help: `dblclick <sel>`
|
package/docs/ELECTRON.md
CHANGED
|
@@ -364,8 +364,8 @@ Electron support is gated by the same release evidence as the rest of the wrappe
|
|
|
364
364
|
|
|
365
365
|
- `RQ-0096` in [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) records the contract, runtime, test, and verification coverage.
|
|
366
366
|
- `electron-lifecycle` and `electron-probe` scenarios in `scripts/agent-browser-efficiency-benchmark.mjs` track the token-efficiency claim deterministically (no real browser, no real launches).
|
|
367
|
-
- Fake-upstream coverage
|
|
368
|
-
- Real-app validation is a manual `tmux` smoke pass per the maintainer notes in `AGENTS.md`; the 2026-05-21 dogfood result
|
|
367
|
+
- Fake-upstream Electron discovery and lifecycle coverage lives in `test/agent-browser.extension-electron-discovery.test.ts` and `test/agent-browser.extension-electron-lifecycle.test.ts`; ref/session recovery coverage lives in `test/agent-browser.extension-ref-guards.test.ts`, with shared validation paths in `test/agent-browser.extension-validation.test.ts`.
|
|
368
|
+
- Real-app validation is a manual `tmux` smoke pass per the maintainer notes in `AGENTS.md`; the 2026-05-21 dogfood result and resulting decision are summarized in the repo-local [`Electron ADR`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/plans/electron-extension-2026-05-20.md).
|
|
369
369
|
|
|
370
370
|
Run the local gate the same way as the rest of the project:
|
|
371
371
|
|
package/docs/RELEASE.md
CHANGED
|
@@ -173,8 +173,8 @@ Maintainer constraints for evolving scenarios and version bumps are summarized u
|
|
|
173
173
|
- no repo-local `.pi/extensions/agent-browser.ts` autoload shim is present
|
|
174
174
|
- `LICENSE` exists in the repo and the packed tarball
|
|
175
175
|
- canonical published docs are present
|
|
176
|
-
- `npm pack --json --dry-run` runs the `
|
|
177
|
-
- GitHub/source installs
|
|
176
|
+
- `npm pack --json --dry-run` runs the package `prepare` build once and packs the compiled `dist/extensions/agent-browser/index.js` entrypoint
|
|
177
|
+
- GitHub/source installs use the same `prepare` build; when Pi installs with `npm install --omit=dev`, `scripts/prepare.mjs` installs source-build dev dependencies with lifecycle scripts disabled before building so Pi can load the ignored compiled entrypoint from a fresh clone
|
|
178
178
|
- the package-level doctor command and capability baseline are present
|
|
179
179
|
- compiled extension runtime files are present, including the split result-rendering modules required by the published facade
|
|
180
180
|
- source-only, agent-only, and superseded docs are absent from the tarball
|
|
@@ -192,7 +192,7 @@ The packaged execution smoke intentionally uses a temporary fake `agent-browser`
|
|
|
192
192
|
Current forbidden packed files include:
|
|
193
193
|
|
|
194
194
|
- `AGENTS.md`
|
|
195
|
-
-
|
|
195
|
+
- internal planning docs under `docs/plans/`
|
|
196
196
|
- `.pi/extensions/agent-browser.ts`
|
|
197
197
|
- TypeScript extension source and other test/repo-only maintenance files
|
|
198
198
|
|
package/docs/REQUIREMENTS.md
CHANGED
|
@@ -81,7 +81,7 @@ Define the product requirements and constraints for `pi-agent-browser-native`.
|
|
|
81
81
|
- Because direct-binary usage is commonly blocked in normal agent sessions, the repo must carry a local command reference for the effective `agent_browser` surface and keep it in sync with upstream changes.
|
|
82
82
|
- Repository verification must include a lightweight command-reference drift check against the targeted installed upstream `agent-browser` version.
|
|
83
83
|
- Published package contents should include the canonical user-facing docs plus `LICENSE`.
|
|
84
|
-
- Published package contents should exclude agent-only and
|
|
84
|
+
- Published package contents should exclude agent-only and internal planning docs such as `AGENTS.md` and `docs/plans/`.
|
|
85
85
|
|
|
86
86
|
### Testing guidance
|
|
87
87
|
|
|
@@ -114,7 +114,7 @@ The design should comfortably support workflows such as:
|
|
|
114
114
|
- Package-manifest behavior matters more than repo-local development wiring.
|
|
115
115
|
- The extension should use official `pi` hooks and package resources where possible.
|
|
116
116
|
- The wrapper should stay thin, with upstream `agent-browser` remaining the source of truth for command semantics.
|
|
117
|
-
- Successful and failed tool outcomes should surface bounded machine-readable fields on Pi-facing `details` (`resultCategory`, `successCategory`, `failureCategory`, optional structured `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`, optional `artifactVerification` with the same shape on successful `batchSteps[]` rows, optional `outputFile`, optional `timeoutPartialProgress`) so agents can branch without parsing prose; stateful commands (`auth`, `cookies`, `storage`, `dialog`, `frame`, `state`) plus other structured diagnostics (for example `network`, `diff`, `trace`, `stream`, `dashboard`, `chat`) and `batch` should redact secret-bearing payloads in model-facing `details.data`, including the compact per-step `batch` roll-up on the parent result (full per-step payloads live on `batchSteps[]`). Dialog/prompt-related timeouts should be bounded with recovery `nextActions`; non-dialog timeouts should prefer best-effort per-step progress and retry payloads when a plan is available; no-op scrolls should expose no-movement state instead of only an upstream success boolean; explicit page/container scroll helpers should expose before/after movement evidence. The contract lives in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), enums and classifier precedence live in `extensions/agent-browser/lib/results/categories.ts` and `contracts.ts
|
|
117
|
+
- Successful and failed tool outcomes should surface bounded machine-readable fields on Pi-facing `details` (`resultCategory`, `successCategory`, `failureCategory`, optional structured `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`, optional `artifactVerification` with the same shape on successful `batchSteps[]` rows, optional `outputFile`, optional `timeoutPartialProgress`) so agents can branch without parsing prose; stateful commands (`auth`, `cookies`, `storage`, `dialog`, `frame`, `state`) plus other structured diagnostics (for example `network`, `diff`, `trace`, `stream`, `dashboard`, `chat`) and `batch` should redact secret-bearing payloads in model-facing `details.data`, including the compact per-step `batch` roll-up on the parent result (full per-step payloads live on `batchSteps[]`). Dialog/prompt-related timeouts should be bounded with recovery `nextActions`; non-dialog timeouts should prefer best-effort per-step progress and retry payloads when a plan is available; no-op scrolls should expose no-movement state instead of only an upstream success boolean; explicit page/container scroll helpers should expose before/after movement evidence. The contract lives in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), enums and classifier precedence live in `extensions/agent-browser/lib/results/categories.ts` and `contracts.ts`, and presentation-time summaries, redaction, network request follow-ups, and artifact verification rollups are assembled in `extensions/agent-browser/lib/results/presentation.ts` (`buildPageChangeSummary`, command taxonomy predicates from `command-taxonomy.ts`, `redactPresentationData`, `buildArtifactVerificationSummary`, `buildBatchPresentation`).
|
|
118
118
|
- User-facing docs belong in `README.md` and the canonical published files under `docs/`.
|
|
119
119
|
- Agent workflow and deeper testing procedures can stay in `AGENTS.md`, but published docs must not depend on that file being present.
|
|
120
120
|
- When upstream `agent-browser` changes, refresh the local command reference, prompt guidance, and other extension-side docs so agents still have a repo-readable equivalent of the blocked direct-binary help path.
|
|
@@ -130,8 +130,3 @@ The design should comfortably support workflows such as:
|
|
|
130
130
|
- Upstream restore-state periodic autosave remains upstream-owned. Forward `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` unchanged, document its 30-second default and `0` disable behavior, and do not duplicate its timer or claim its restore files as wrapper artifacts.
|
|
131
131
|
- Read-only upstream `skills list`, `skills get …`, and `skills path …` must stay free of implicit managed `--session` under default `sessionMode: "auto"` (still with `--json`), matching plain-text `--help` / `--version` inspection semantics so bundled skill text does not pin or rotate the active browser session; new `skills` subcommands pick up that behavior only after allowlisting in `extensions/agent-browser/lib/runtime.ts` with regression coverage.
|
|
132
132
|
- Optional `semanticAction.session` on native `agent_browser` must compile to a leading `--session <name>` pair before upstream `find` or `select` argv so the shorthand can target a named upstream browser without hand-built `args`, while `buildExecutionPlan` still skips double-injecting the extension-managed implicit session whenever planned argv already starts with `--session`; stale-ref retries for compiled `find` actions and bounded `try-*` candidate `nextActions` must preserve that same prefix. Contract in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#semanticaction) / [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#sessionmode); implementation in `extensions/agent-browser/index.ts` and `extensions/agent-browser/lib/runtime.ts`.
|
|
133
|
-
|
|
134
|
-
## Open design questions
|
|
135
|
-
|
|
136
|
-
- How much session convenience should the extension add by default versus leaving explicit session naming entirely to upstream `agent-browser` semantics?
|
|
137
|
-
- Exactly which high-value result renderers belong in v1 beyond screenshots/images and a few compact summaries?
|
package/docs/SUPPORT_MATRIX.md
CHANGED
|
@@ -26,15 +26,15 @@ When upstream ships a new `agent-browser` or the inventory changes:
|
|
|
26
26
|
|
|
27
27
|
## Audit result
|
|
28
28
|
|
|
29
|
-
- Target upstream: `agent-browser 0.32.
|
|
29
|
+
- Target upstream: `agent-browser 0.32.2` (must match `CAPABILITY_BASELINE.targetVersion` in [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs)).
|
|
30
30
|
- Source of truth: `CAPABILITY_BASELINE.inventorySections` in the same file (stable `id` keys: `skills`, `core-commands`, `state-tabs-frames-dialogs`, `network-storage-artifacts-diagnostics`, `batch-auth-setup-ai`, `options-and-env`).
|
|
31
|
-
- Status: source and wrapper adaptation are complete for the 2026-07-
|
|
32
|
-
- High-priority support gaps: 2026-05-26 audit found sessionless local commands and command-scoped value flags needed sharper wrapper handling; runtime/tests/docs now cover those paths. The 0.28.0 rebaseline added local `mcp` and `plugin` surfaces plus plugin-backed credential login; wrapper docs/tests mark `mcp` and known `plugin` commands sessionless, with no compatibility shim for older upstream releases. The 0.29.1 rebaseline added upstream `@agent-browser/sandbox` helper-package guidance and stricter `install --with-deps` failure semantics; no new wrapper runtime mode or bundled dependency was required. The 0.30.1 rebaseline fixed upstream `wait --url` glob matching, so constrained `job.assertUrl` delegates glob and exact patterns directly to `wait --url`. The 0.31.0 rebaseline adds restore workflow and namespace/session lifecycle surfaces (`--restore`, restore checks, `--namespace`, `session id`, and `session info`) without adding a wrapper compatibility layer. The 0.31.1 rebaseline fixes upstream React renderer selection for `react tree`, `react inspect`, and `react suspense`; no wrapper runtime change was required. The 0.31.2 rebaseline adds the local-launch-only `--webgpu` preset and `doctor --webgpu`, plus periodic restore-state autosaves controlled by `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS`; the wrapper treats WebGPU as launch-scoped and leaves autosave ownership upstream. The 0.32.0 rebaseline hardens request/worker/popup/WebRTC containment behind `--allowed-domains`, fixes waits against already-complete documents, and adds a separate `@agent-browser/eve` package; the wrapper makes containment launch-scoped, retains its final-URL check as defense in depth, and adds no Eve-specific mode. Remaining upstream-owned caveat: current help still mentions `wait <selector> --state hidden` / `detached` and `find ... uncheck`, but runtime probes show those advertised shapes still fail, so wrapper docs keep `wait --fn` predicates and direct `uncheck` passthrough guidance.
|
|
31
|
+
- Status: source and wrapper adaptation are complete for the 2026-07-17 0.32.2 audit; the 0.32.1–0.32.2 changes stay isolated to the separate `@agent-browser/eve` package, while the current capability audit closes the prior missing `read [url]` inventory/presentation path without adding a native Pi input mode.
|
|
32
|
+
- High-priority support gaps: 2026-05-26 audit found sessionless local commands and command-scoped value flags needed sharper wrapper handling; runtime/tests/docs now cover those paths. The 0.28.0 rebaseline added local `mcp` and `plugin` surfaces plus plugin-backed credential login; wrapper docs/tests mark `mcp` and known `plugin` commands sessionless, with no compatibility shim for older upstream releases. The 0.29.1 rebaseline added upstream `@agent-browser/sandbox` helper-package guidance and stricter `install --with-deps` failure semantics; no new wrapper runtime mode or bundled dependency was required. The 0.30.1 rebaseline fixed upstream `wait --url` glob matching, so constrained `job.assertUrl` delegates glob and exact patterns directly to `wait --url`. The 0.31.0 rebaseline adds restore workflow and namespace/session lifecycle surfaces (`--restore`, restore checks, `--namespace`, `session id`, and `session info`) without adding a wrapper compatibility layer. The 0.31.1 rebaseline fixes upstream React renderer selection for `react tree`, `react inspect`, and `react suspense`; no wrapper runtime change was required. The 0.31.2 rebaseline adds the local-launch-only `--webgpu` preset and `doctor --webgpu`, plus periodic restore-state autosaves controlled by `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS`; the wrapper treats WebGPU as launch-scoped and leaves autosave ownership upstream. The 0.32.0 rebaseline hardens request/worker/popup/WebRTC containment behind `--allowed-domains`, fixes waits against already-complete documents, and adds a separate `@agent-browser/eve` package; the wrapper makes containment launch-scoped, retains its final-URL check as defense in depth, and adds no Eve-specific mode. The 0.32.1–0.32.2 rebaseline updates only eve compatibility/packaging, but the full current-surface audit also found the local reference had omitted upstream `read [url]` since 0.30.0; baseline/help sampling, content-first presentation, tab-target preservation, timeout budgeting, docs, and tests now cover it. Remaining upstream-owned caveat: current help still mentions `wait <selector> --state hidden` / `detached` and `find ... uncheck`, but runtime probes show those advertised shapes still fail, so wrapper docs keep `wait --fn` predicates and direct `uncheck` passthrough guidance.
|
|
33
33
|
- Post-`v0.2.29` review state: commits `eb55320` through `86abbfb` add browser guidance/smoke coverage plus `RQ-0086` click-probe reduction, `RQ-0087` same-snapshot form fill batching, `RQ-0088` current-ref fallback on locator misses, `RQ-0089` direct-upstream click mutation investigation, and `RQ-0090` stop-boundary/artifact-path guidance. Verification gates below were rerun on 2026-05-18 after those tasks landed. Constrained `job` (`RQ-0064`), the lightweight `qa` preset (`RQ-0065`), the experimental `sourceLookup` helper (`RQ-0066`), the experimental `networkSourceLookup` helper (`RQ-0067`), optional Exa/Brave-backed `agent_browser_web_search` with Pi-scoped package config (`RQ-0121`), and agent recovery for search/profile configuration failures (`RQ-0122`) are implemented; see [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#job), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#qa), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#sourcelookup), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#networksourcelookup), and [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#optional-companion-web-search). Reusable browser recipes (`RQ-0068`) are intentionally not adopted as a runtime surface; see [`ARCHITECTURE.md`](ARCHITECTURE.md#no-reusable-recipe-layer-yet).
|
|
34
34
|
|
|
35
|
-
##
|
|
35
|
+
## UX/reliability decisions from 2026-05-29 agent feedback
|
|
36
36
|
|
|
37
|
-
|
|
37
|
+
Closed feedback decisions map to their canonical contracts in [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md). Keep this active matrix limited to release-critical status and gates.
|
|
38
38
|
|
|
39
39
|
Current summary:
|
|
40
40
|
|
|
@@ -42,24 +42,25 @@ Current summary:
|
|
|
42
42
|
| --- | --- | --- |
|
|
43
43
|
| RQ-0110–RQ-0120 | Agent feedback triage resolved or documented; remaining unsupported areas are environment/upstream-owned. | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
44
44
|
| RQ-0123–RQ-0127 | Stress-report wrapper fixes shipped; prompt-derived business-action blocking remains intentionally out of scope. | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
45
|
-
|
|
|
45
|
+
| `agent-browser 0.27.2` baseline | Historical upstream rebaseline shipped; `RQ-0101` remains the compact-snapshot high-value-controls decision. | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
46
46
|
| RQ-0128 | Upstream `agent-browser 0.27.3` install-only rebaseline shipped; no new wrapper capability adopted. | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
47
47
|
| RQ-0129 | Upstream `agent-browser 0.29.1` rebaseline shipped; sandbox helpers are documented upstream package guidance, not a wrapper runtime. | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
48
48
|
| RQ-0130 | Upstream `agent-browser 0.30.1` rebaseline shipped; `wait --url` glob support replaces the wrapper `job.assertUrl` predicate workaround. | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
49
49
|
| RQ-0131 | Upstream `agent-browser 0.31.0` rebaseline shipped; restore workflow and namespace/session lifecycle globals are parsed, documented, and carried through wrapper-managed probes/state. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0310-rebaseline) |
|
|
50
50
|
| RQ-0132 | Upstream `agent-browser 0.31.1` rebaseline shipped; React renderer selection is upstream-fixed for `react tree`, `react inspect`, and `react suspense`, with no wrapper CLI/schema changes. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0311-rebaseline) |
|
|
51
51
|
| RQ-0133 | Upstream `agent-browser 0.31.2` rebaseline adds WebGPU launch/doctor/MCP surfaces and periodic restore-state autosaves; wrapper launch parsing/policy and docs now cover both. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0312-rebaseline) |
|
|
52
|
-
| RQ-0134 | Upstream `agent-browser 0.32.0` rebaseline hardens domain containment, fixes completed-page waits, and adds a separate
|
|
52
|
+
| RQ-0134 | Upstream `agent-browser 0.32.0` rebaseline hardens domain containment, fixes completed-page waits, and adds a separate eve extension package; wrapper launch policy and docs cover the relevant behavior. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0320-rebaseline) |
|
|
53
|
+
| RQ-0135 | Upstream `agent-browser 0.32.2` rebaseline updates eve packaging only and closes the wrapper's prior missing `read [url]` inventory/presentation contract. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0322-rebaseline) |
|
|
53
54
|
|
|
54
55
|
## Verification evidence
|
|
55
56
|
|
|
56
|
-
Re-run the gates below before each release; this table records what the closure audit exercised. The 0.32.
|
|
57
|
+
Re-run the gates below before each release; this table records what the closure audit exercised. The 0.32.2 / Pi 0.80.10 local rebaseline gates passed on 2026-07-17; the prior 0.32.0 / Pi 0.80.9 release-composition and platform rows remain explicitly historical evidence.
|
|
57
58
|
|
|
58
59
|
| Gate | Evidence | Status |
|
|
59
60
|
| --- | --- | --- |
|
|
60
|
-
| Default local gate | `npm run verify` checks generated playbook drift, clean-builds generated `dist/`, runs `tsc --noEmit`, unit/fake tests, generated command-reference blocks, and live command-reference sampling. | **Current for 0.32.
|
|
61
|
+
| Default local gate | `npm run verify` checks generated playbook drift, clean-builds generated `dist/`, runs `tsc --noEmit`, unit/fake tests, generated command-reference blocks, and live command-reference sampling. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-17 (585 passed, 2 opt-in skips; live command-reference verification passed). |
|
|
61
62
|
| Pre-PR local gate | `npm run verify -- pre-pr` composes the default gate with package-content verification. Use before larger local handoffs or PR-ready claims when lifecycle/platform/live dogfood cost is not warranted. | **Current for 0.32.0:** covered by the stronger passing release composition on 2026-07-16; orchestration remains locked by `test/project-verify.test.ts`. |
|
|
62
|
-
| Real upstream contract | `npm run verify -- real-upstream` runs the localhost fixture matrix against the real installed `agent-browser` matching the baseline. | **Current for 0.32.
|
|
63
|
+
| Real upstream contract | `npm run verify -- real-upstream` runs the localhost fixture matrix against the real installed `agent-browser` matching the baseline. | **Current for 0.32.2:** pass on 2026-07-17 (2/2 real-upstream tests), including visible `read` content against the localhost contract fixture; the known CSS-selector click probe constraint remains documented in the fixture. |
|
|
63
64
|
| Packaged Pi smoke | `npm run verify -- package-pi` validates package contents, loads the packaged `agent_browser` tool without requiring optional Brave config, and executes fake-upstream `--version`. | **Current for 0.32.0 / package 0.2.68 / Pi 0.80.9:** pass on 2026-07-16 inside `npm run verify -- release` (118 packed entries; exactly one packaged `agent_browser`; invocation passed). |
|
|
64
65
|
| Startup profile | `npm run verify -- startup-profile --samples <n>` clean-builds generated `dist/`, records direct package entrypoint import/factory timing in fresh Node processes, and writes `.artifacts/startup-profile/latest.json`. It must not launch Pi, tmux, mise, npm, browsers, or `agent-browser`; full Pi TUI ready-prompt profiling is intentionally excluded after it proved too invasive for routine verification. Run this opt-in evidence when package layout, the compiled entrypoint, top-level imports, schema registration, or prompt/config startup logic changes. | **Current for compiled entrypoint:** pass on 2026-06-21 (`npm run verify -- startup-profile --samples 3`; direct compiled entrypoint import+factory median 47.3 ms, below the 250 ms budget). Full-Pi startup numbers from the unsafe tmux profiler are not accepted as ongoing release evidence. |
|
|
65
66
|
| Deterministic dogfood smoke | `npm run verify -- dogfood` (`scripts/verify-agent-browser-dogfood.ts`) drives the native wrapper against a local file fixture through top-level `qa`, `semanticAction`, constrained `job`, screenshot artifact verification, and session close with the real `agent-browser` on `PATH`. | **Current for 0.32.0:** pass on 2026-07-15; the packaged target-local suite also passed on macOS, Ubuntu, and native Windows in the Pi 0.80.9 release matrix on 2026-07-16. |
|
|
@@ -68,7 +69,7 @@ Re-run the gates below before each release; this table records what the closure
|
|
|
68
69
|
| Crabbox platform smoke | `npm run check:platform-smoke` syntax-checks the harness and cheap invariants. `npm run smoke:platform:ubuntu-image` builds the project-owned Linux image, `npm run smoke:platform:doctor` checks Crabbox 0.26.0+ and local target readiness, and `npm run smoke:platform:all` runs doctor first, then fast target-local `platform-build` (`npm run verify -- platform-target`, pack, clean Pi install) plus `browser-dogfood-smoke` on Crabbox `macos`, `ubuntu`, and `windows-native`; see [`platform-smoke.md`](platform-smoke.md). Target artifacts include Crabbox/provider/work-root metadata, and release review also checks provider-specific `crabbox list` commands for leftover leases/clones. | **Current for 0.32.0 / Pi 0.80.9:** pass on 2026-07-16 inside `npm run verify -- release`; evidence roots are `run-1784229255840-b25cii` (macOS), `run-1784229255842-qhl9z7` (Ubuntu), and `run-1784229255843-ez04e8` (native Windows). The canonical Windows snapshot is the power-off `crabbox-ready` baseline with agent-browser 0.32.0; no audit-owned leases/clones remained. |
|
|
69
70
|
| `verify -- release` / `prepublishOnly` | `npm run verify -- release` chains the default gate with the configured-source lifecycle harness, packaged Pi smoke, and the release-blocking Crabbox platform matrix (`verifySteps` `release` in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs)). `package.json` `prepublishOnly` runs that compose before `npm pack --dry-run` during `npm publish`. It intentionally omits standalone real-upstream, host-only dogfood, and benchmark modes—see [`RELEASE.md`](RELEASE.md#pre-release-checks). | **Current for 0.32.0 / package 0.2.68 / Pi 0.80.9:** pass on 2026-07-16, including default unit/fake/docs/typecheck, live command-reference sampling, lifecycle, packaged Pi smoke, and the required three-target matrix. |
|
|
70
71
|
| Configured-source lifecycle | `npm run verify -- lifecycle` (`scripts/verify-lifecycle.mjs`) drives `/reload`, closes and relaunches Pi with the same exact `--session-id`, checks the JSONL session header id, session continuity, slash-command sentinel tokens (`v1` before reload and `v2` after full relaunch because compiled JS package modules are process-cached), persisted spill reachability, and real Pi `tool_result` failure-patch semantics for a QA reclassification with a fake upstream on `PATH`. Default Pi model is `zai/glm-5.2`; default per-step wait is **180000 ms** (`DEFAULT_TIMEOUT_MS`); override model with `--model <id>` and waits with `--timeout-ms <ms>`. Passthrough flags in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs): `--keep-artifacts`, `--model`, `--verbose`, and `--timeout-ms` plus a value (for example `npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal --keep-artifacts --verbose --timeout-ms 600000`). | **Current for 0.32.0 / Pi 0.80.9:** pass on 2026-07-16 inside `npm run verify -- release`; reload/relaunch continuity, persisted spill reachability, and failure-patch assertions passed. |
|
|
71
|
-
| Quick isolated Pi smoke | `pi --approve --no-extensions --no-skills -e . --tools agent_browser` from trusted repo root; native `agent_browser` only. | **Current for 0.
|
|
72
|
+
| Quick isolated Pi smoke | `pi --approve --no-extensions --no-skills -e . --tools agent_browser` from trusted repo root; native `agent_browser` only. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-17 via tmux with `zai/glm-5.2:high`; checkout-loaded `read https://example.com` rendered the `Example Domain` body rather than only the URL, and `close` returned `closed: true`. The disposable Pi session directory and tmux session were removed. Broader Sauce Demo checkout/artifact evidence remains historical from 0.29.1 / Pi 0.79.9. |
|
|
72
73
|
|
|
73
74
|
Runtime floor note: package metadata keeps Pi core package peer ranges wildcard per installed Pi package docs, but `pi-agent-browser-doctor` / `npm run doctor` treats `pi --version` below 0.80.6 as a setup failure. `npm run doctor` passed against Pi 0.80.6 on 2026-07-11. This keeps package dependency shape aligned with Pi package loading while still making unsupported host Pi versions a release and first-run blocker.
|
|
74
75
|
|
|
@@ -77,7 +78,7 @@ Runtime floor note: package metadata keeps Pi core package peer ranges wildcard
|
|
|
77
78
|
| Baseline section | Baseline items | Documentation | Runtime handling | Test coverage | Validation status |
|
|
78
79
|
| --- | --- | --- | --- | --- | --- |
|
|
79
80
|
| Built-in skills | 15 canonical tokens from baseline section `skills`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#built-in-skills). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#built-in-skills), generated baseline block, README proof section, release docs. | `needsManagedSession` keeps read-only skills inspection sessionless while preserving thin upstream passthrough; upstream `@agent-browser/sandbox` remains external package guidance, not a bundled wrapper dependency. | Runtime and extension-validation skills/provider matrix; real-upstream inspection/skills group. | Supported. |
|
|
80
|
-
| Core page, element, navigation, and extraction commands |
|
|
81
|
+
| Core page, element, navigation, and extraction commands | 81 canonical tokens from baseline section `core-commands`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#core-page-and-element-commands). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#core-page-and-element-commands), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md), README quick start. | Thin passthrough with wrapper-owned JSON/session planning, ref guidance, artifact verification, page-change summaries, click-dispatch diagnostics, no-op scroll/focus diagnostics, shorthand compilers, and redaction. | Real-upstream core matrix plus fake core matrix for passthrough, ordering, diagnostics, and compiler validation. | Supported. Upstream semantics remain upstream-owned. |
|
|
81
82
|
| Sessions, state, tabs, frames, dialogs, and windows | 24 canonical tokens from baseline section `state-tabs-frames-dialogs`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#session-state-frames-dialogs-windows-and-inspection-commands). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#session-state-frames-dialogs-windows-and-inspection-commands), stateful workflow notes, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details). | Stateful summaries/redaction, state artifact handling, sessionless local command planning, managed-session restore, tab target pinning, and close alias cleanup. | Extension-validation stateful matrix, runtime session/resume tests, presentation redaction tests, lifecycle harness. | Supported. External profile/auth state remains operator-owned. |
|
|
82
83
|
| Network, storage, artifacts, diagnostics, and performance | 43 canonical tokens from baseline section `network-storage-artifacts-diagnostics`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#page-state-finding-mouse-settings-network-and-storage). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#page-state-finding-mouse-settings-network-and-storage), diagnostic sections, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details). | Thin passthrough plus compact diagnostics, route-mock warnings, useful-but-redacted storage output, stream idempotency normalization, artifact metadata, missing-ffmpeg warnings, sensitive-data redaction, timeout bounds, and cleanup-pair guidance. | Fake non-core matrix and safe real-upstream coverage for network/HAR, diff, trace/profiler, console/errors/highlight, stream, vitals, and React missing-renderer. | Supported. Environment-sensitive operations need suitable local/browser state. |
|
|
83
84
|
| Batch, auth, confirmations, setup, dashboard, devices, and AI commands | 33 canonical tokens from baseline section `batch-auth-setup-ai`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#batch-auth-confirmations-sessions-chat-dashboard-devices-and-setup). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#batch-auth-confirmations-sessions-chat-dashboard-devices-and-setup), README security notes, release docs. | Native-tool batch stdin, generated `job`/`qa`/lookup batch plans, auth/confirmation redaction, sessionless local auth/setup/dashboard/doctor/plugin planning, plugin list/show JSON envelope normalization, bare-`mcp` validation with `mcp --help` preserved, timeout/cleanup guidance. | Parser/runtime plugin and MCP unit coverage; fake-upstream plugin list/show and MCP help/blocking coverage; real-upstream plugin list shape probe; structured input-mode tests; efficiency benchmark scenarios. | Supported. Interactive side-effecting setup/auth/chat remains upstream-owned. `plugin` is local/sessionless; `mcp` is external-client-only except help; `auth login --credential-provider` resolves credentials via a plugin; `install --with-deps` failures remain upstream-owned. |
|
|
@@ -85,9 +86,9 @@ Runtime floor note: package metadata keeps Pi core package peer ranges wildcard
|
|
|
85
86
|
|
|
86
87
|
## Follow-up decision after closure
|
|
87
88
|
|
|
88
|
-
|
|
89
|
+
The compact decision index lives at [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md). Keep this section as the active index of shipped follow-up areas and their canonical contracts.
|
|
89
90
|
|
|
90
|
-
| Area | Active contract |
|
|
91
|
+
| Area | Active contract | Decision index |
|
|
91
92
|
| --- | --- | --- |
|
|
92
93
|
| Native structured input modes (`job`, `qa`, `sourceLookup`, `networkSourceLookup`, `semanticAction`) | [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md), [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md) | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
|
93
94
|
| Electron lifecycle, manual CDP attach, desktop readiness, and tab/surface recovery | [`ELECTRON.md`](ELECTRON.md), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#electron), [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#electron-desktop-apps) | [`docs/support-notes.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/support-notes.md) |
|
package/docs/TOOL_CONTRACT.md
CHANGED
|
@@ -165,7 +165,7 @@ The extension always plans normal browser commands with `--json` prepended in `e
|
|
|
165
165
|
- For dialogs and frames, use dialog status/accept/dismiss and frame <selector|main> through native args; dialog commands and eval snippets that look like alert/confirm/prompt/dialog triggers are shorter-bounded than normal browser calls, and timed-out dialog-like interactions may add inspect-dialog-after-timeout, dismiss-dialog-after-timeout, or recover-fresh-session-after-dialog-timeout nextActions. When --confirm-actions produces a pending confirmation, use details.nextActions or exact confirm <id> / deny <id> calls instead of inventing ids.
|
|
166
166
|
- If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or qa.attached; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait <selector> --state hidden. Use electron.probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort: use explicit --timeout or top-level timeoutMs for legitimately slow waits, and treat a successful payload like "waited":"timeout" as elapsed time only—verify completion with an observed condition, fresh snapshot, or screenshot.
|
|
167
167
|
- For feed, timeline, or inbox reading tasks, focus on the main timeline/list region and read the first item there rather than unrelated composer or sidebar content.
|
|
168
|
-
- For read-only browsing tasks,
|
|
168
|
+
- For read-only browsing tasks, use read <url> for documentation or other unstructured text without launching Chrome, or read with no URL for rendered active-tab DOM. Prefer the current snapshot, structured ref labels, getters, or scoped eval --stdin when you need interactive structure or targeted page state. Only click into media viewers, detail routes, or new pages when the current view does not contain the needed information.
|
|
169
169
|
- For downloads, prefer download <selector> <path> when an element click should save a file; simple loopback anchor downloads are saved to the requested path when the wrapper can resolve an HTTP(S) href. Do not rely on click alone when you need the downloaded file on disk.
|
|
170
170
|
- On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or semanticAction/job select) instead of clicking option refs; for custom comboboxes, a click/semanticAction may only focus the field, so re-snapshot and fall back to type, press Enter/arrow keys, or visible option refs.
|
|
171
171
|
- When using eval --stdin, scope checks and actions to the target element or route whenever possible instead of relying on broad page-wide text heuristics.
|
|
@@ -252,7 +252,7 @@ If a raw `find` or compiled `semanticAction` fails with `failureCategory: "selec
|
|
|
252
252
|
|
|
253
253
|
If a compiled `semanticAction` fails with `failureCategory: "selector-not-found"`, visible content can also include an `Agent-browser candidate fallbacks` block when the wrapper has bounded role/name retries for that locator and action, and `details.nextActions` includes the normal `refresh-interactive-refs` snapshot step plus those entries. When `session` was provided, candidate retry args preserve the same `--session <session>` prefix. Today `buildSemanticActionCandidateActions` in `extensions/agent-browser/index.ts` only appends click candidates for `click` + `text` → `try-button-name-candidate` and `try-link-name-candidate`. Fill misses no longer emit `find … fill <text>` retry actions because those would repeat potentially sensitive text. Instead, when the same selector-miss snapshot finds exact current editable refs (`searchbox` or `textbox`), the wrapper emits `details.richInputRecovery`, visible `Rich input recovery`, and `focus-current-editable-ref` / `click-current-editable-ref` (numbered when ambiguous) next actions. Those actions carry only focus/click argv for the candidate ref; they do not copy fill text, press `Enter`, or submit. Use `keyboard inserttext` or `keyboard type` with the intended text only after focusing the right current ref, and submit only when the user flow explicitly calls for it. Candidate fallbacks are heuristics, not proof that an element exists; inspect the page when several controls could share the same name.
|
|
254
254
|
|
|
255
|
-
If a compiled `semanticAction` `find` action fails with `failureCategory: "stale-ref"`, `details.nextActions` includes `retry-semantic-action-after-stale-ref` with the same redacted compiled argv as `details.compiledSemanticAction` in `params.args` (any leading `--session` pair from `semanticAction.session`, then the `find` tokens). The wrapper appends that entry **after** any `refresh-interactive-refs` snapshot step from `buildAgentBrowserNextActions` in `extensions/agent-browser/lib/results/action-recommendations.ts` (
|
|
255
|
+
If a compiled `semanticAction` `find` action fails with `failureCategory: "stale-ref"`, `details.nextActions` includes `retry-semantic-action-after-stale-ref` with the same redacted compiled argv as `details.compiledSemanticAction` in `params.args` (any leading `--session` pair from `semanticAction.session`, then the `find` tokens). The wrapper appends that entry **after** any `refresh-interactive-refs` snapshot step from `buildAgentBrowserNextActions` in `extensions/agent-browser/lib/results/action-recommendations.ts` (see `extensions/agent-browser/index.ts` where `nextActions` is merged). That retry is only offered because the semantic target is stable and the stale-ref error proves the previous action did not execute; `select` shorthands with stale `@e…` selectors and direct stale `@e…` commands still return refresh guidance instead of an unsafe blind retry.
|
|
256
256
|
|
|
257
257
|
For direct page-scoped `@e…` refs, successful `snapshot` results record `details.refSnapshot` with the latest ref ids and page target for the session. A failed session `snapshot` whose upstream error says `No active page` clears that session’s prior ref snapshot and records `details.refSnapshotInvalidation.reason: "no-active-page"`; mutation-prone `@e…` preflight then fails with `failureCategory: "stale-ref"` until a later successful `snapshot -i` records fresh refs. Before mutation-prone ref commands such as `click`, `fill`, `check`, `select`, `download`, drag/upload/keyboard-style actions, or equivalent batch steps run, the wrapper rejects refs from an older page target, refs absent from the latest same-page snapshot, or refs from an invalidated snapshot state. This is a best-effort wrapper guard against upstream ref-number recycling after navigation; it does not prove the DOM stayed unchanged after the snapshot. Refresh with the session-aware `refresh-interactive-refs` next action before retrying.
|
|
258
258
|
|
|
@@ -713,7 +713,7 @@ Ref preflight details (command taxonomy in `extensions/agent-browser/lib/command
|
|
|
713
713
|
|
|
714
714
|
**Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with `redactStructuredPresentationValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials`, and scrub other argv tokens for URLs and inline secrets. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs.
|
|
715
715
|
|
|
716
|
-
`nextActions` is an optional machine-readable list of exact native `agent_browser` follow-ups. Each entry includes `tool: "agent_browser"`, an `id`, a short `reason`, optional `safety`, and either `params` (`args`, optional `stdin`, optional `sessionMode`, optional `networkSourceLookup`, optional `electron`) or an `artifactPath` for saved-file workflows. Agents should prefer these payloads over prose when present. When a result ran under upstream `--namespace <name>`, follow-up `params.args` preserve that namespace so retries and diagnostics target the same daemon/restore-state space. Tab/session recovery id strings are centralized in `AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, while rich-input focus/click recovery ids are centralized in `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS` plus `getAgentBrowserRichInputRecoveryNextActionId(s)` in `extensions/agent-browser/lib/results/recovery-actions.ts
|
|
716
|
+
`nextActions` is an optional machine-readable list of exact native `agent_browser` follow-ups. Each entry includes `tool: "agent_browser"`, an `id`, a short `reason`, optional `safety`, and either `params` (`args`, optional `stdin`, optional `sessionMode`, optional `networkSourceLookup`, optional `electron`) or an `artifactPath` for saved-file workflows. Agents should prefer these payloads over prose when present. When a result ran under upstream `--namespace <name>`, follow-up `params.args` preserve that namespace so retries and diagnostics target the same daemon/restore-state space. Tab/session recovery id strings are centralized in `AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, while rich-input focus/click recovery ids are centralized in `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS` plus `getAgentBrowserRichInputRecoveryNextActionId(s)` in `extensions/agent-browser/lib/results/recovery-actions.ts`; docs and tests mirror those registries/helpers rather than inventing recovery ids in prose. Current recommendations include: timed-out jobs/batches with a retryable read-only/idempotent first incomplete step → `retry-timeout-step`, while timed-out flows whose first incomplete step may be mutating → `inspect-current-page-after-timeout` (`snapshot -i`) before splitting the remaining work into shorter batches; raw `connect` success → session-scoped `list-connected-session-tabs` only, then the agent should inspect/select a stable `tab t<N>` target and run `snapshot -i` explicitly; `snapshot` failures whose upstream error says `No active page` and whose wrapper result has a known session → `list-tabs-after-no-active-page` only, because this path has no wrapper-observed safe tab id to select atomically; browser profile/user-data-dir resolution failures → `inspect-browser-profiles` (`profiles`) and `run-agent-browser-doctor` (`doctor`) before retrying opens; Electron launches → wrapper-tracked `electron.status` / `electron.probe` / `electron.cleanup` actions plus session-scoped tab/snapshot inspection when attached; Electron status/probe mismatch diagnostics → `reattach-electron-launch` plus fresh tab/snapshot inspection; Electron post-command health failures → status/probe/cleanup for the same `launchId`; Electron or contenteditable fill verification mismatches → `inspect-after-fill-verification` and `verify-filled-value`; Electron same-URL ref freshness warnings → `refresh-electron-refs-after-rerender`; packaged-Electron `sourceLookup` no-candidate diagnostics → session snapshot, launch probe, and tab list; Electron cleanup partial failures → status plus retry-cleanup for the same wrapper-owned `launchId`; `open` success → `snapshot -i`; mutating/navigation commands (see `buildAgentBrowserNextActions` in source for the exact command set) → `snapshot -i`; stale refs and selector failures → `snapshot -i` via `refresh-interactive-refs` (prefixed with `--session <name>` when the failed call ran in a named or managed session); selector misses with exact current snapshot role/name matches → direct ref retries via `try-current-visible-ref` or bounded `try-current-visible-ref-N` for non-fill targets; semantic `fill` selector misses with exact current editable refs → `focus-current-editable-ref` / `click-current-editable-ref` or numbered variants that do not include fill text or submit; unknown getter shortcuts such as `title` / `url` → exact read-only retries like `get title` / `get url` with ids `use-get-title` / `use-get-url`; compact `network requests` results with safe request IDs → bounded read-only request detail, `networkSourceLookup`, path filter, or HAR-capture follow-ups; semantic `selector-not-found` failures that compiled from `semanticAction` may append `try-button-name-candidate` or `try-link-name-candidate` after presentation `nextActions` only for the bounded click pair enumerated under `semanticAction`; semantic `stale-ref` failures that compiled from `semanticAction` `find` argv may also include `retry-semantic-action-after-stale-ref` after that snapshot step; successful snapshots or qualifying same-URL non-Electron top-level clicks (see `overlayBlockers` below) with snapshot evidence of likely overlay/banner/dialog close controls may append `inspect-overlay-state` and bounded `try-overlay-blocker-candidate-*` entries; successful top-level `scroll` calls whose pre/post viewport and sampled scroll-container positions do not change may append `inspect-after-noop-scroll` and `verify-noop-scroll-visually`; explicit combobox-targeted actions that focus a combobox without visible options may append `inspect-focused-combobox`, `try-open-combobox-with-arrow`, and `try-open-combobox-with-enter`; `get text <selector>` calls with hidden/multiple CSS matches may append `inspect-visible-text-candidates` with a read-only `eval --stdin` probe (each prefixed with `--session <name>` when `details.sessionName` is set, same `sessionPrefixArgs` rule as other session-scoped follow-ups); confirmations → exact `confirm <id>` and `deny <id>` choices; generic tab drift → `list-tabs-for-recovery` with `tab list` first, then select or confirm the stable target before running `snapshot -i`; about:blank or tab-drift recovery with a wrapper-known target → `list-tabs-for-about-blank-recovery` or `list-tabs-for-tab-drift-recovery`, plus `select-intended-tab-after-drift` and `snapshot-after-tab-recovery` when the wrapper already observed the stable `t<N>` tab id; `wait --text` assertion failures → `inspect-after-text-assertion-failure` with a read-only snapshot; download verification failures or missing successful download artifacts → `wait --download [path]`; saved artifacts → the artifact path to inspect/consume after checking `artifactVerification`/metadata; missing non-download artifacts → `verify-artifact-path` so agents do not trust an absent file. When nothing applies, the field is omitted.
|
|
717
717
|
|
|
718
718
|
**Unknown-command getter hints (failure presentation):** `buildErrorPresentation` in `extensions/agent-browser/lib/results/presentation/errors.ts` only runs this path when upstream error text (after model-facing redaction) matches `unknown command`, `unknown subcommand`, or `unrecognized command` (case-insensitive) **and** the failed invocation’s primary command token is one of `attr`, `count`, `html`, `text`, `title`, `url`, or `value`. Visible text then includes a grouped-`get` hint line plus per-token guidance (`get text <selector>`, `get html …`, `get attr …`, `get count …`, `get value …`, `get title`, `get url`). Machine `nextActions` with ids `use-get-title` / `use-get-url` are emitted only for `title` / `url`, with `params.args` optionally prefixed by `--session <name>` when the failed call targeted a named session. If the error string already contains `Agent-browser hint:` from selector recovery (stale-ref or unsupported selector dialect appendages), the getter block is skipped so two stacked `Agent-browser hint:` headers are not emitted.
|
|
719
719
|
|
|
@@ -777,7 +777,7 @@ When `semanticAction` produced compiled `find` argv and the unified result is `f
|
|
|
777
777
|
|
|
778
778
|
Implementation and precedence:
|
|
779
779
|
|
|
780
|
-
- Shared machine-readable types are centralized in `extensions/agent-browser/lib/results/contracts.ts` (including re-exports such as `AgentBrowserNextAction` from `next-actions.ts`). Classifiers live in `categories.ts` (`classifyAgentBrowserSuccessCategory`, `classifyAgentBrowserFailureCategory`, `buildAgentBrowserResultCategoryDetails`—the last prefers an explicit `failureCategory` when the caller already knows the bucket, otherwise it runs the classifier). Generic follow-up assembly lives in `action-recommendations.ts` (`buildAgentBrowserNextActions`). Tab/session recovery ids live in `recovery-actions.ts` (`AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS`, `getAgentBrowserRichInputRecoveryNextActionId`, `getAgentBrowserRichInputRecoveryNextActionIds`, `buildRecoveryNextActions`) and session-aware wrappers live in `recovery-next-actions.ts`. Selector miss and rich-input diagnostic shapes/actions live in `selector-recovery.ts`.
|
|
780
|
+
- Shared machine-readable types are centralized in `extensions/agent-browser/lib/results/contracts.ts` (including re-exports such as `AgentBrowserNextAction` from `next-actions.ts`). Classifiers live in `categories.ts` (`classifyAgentBrowserSuccessCategory`, `classifyAgentBrowserFailureCategory`, `buildAgentBrowserResultCategoryDetails`—the last prefers an explicit `failureCategory` when the caller already knows the bucket, otherwise it runs the classifier). Generic follow-up assembly lives in `action-recommendations.ts` (`buildAgentBrowserNextActions`). Tab/session recovery ids live in `recovery-actions.ts` (`AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS`, `getAgentBrowserRichInputRecoveryNextActionId`, `getAgentBrowserRichInputRecoveryNextActionIds`, `buildRecoveryNextActions`) and session-aware wrappers live in `recovery-next-actions.ts`. Selector miss and rich-input diagnostic shapes/actions live in `selector-recovery.ts`. Failed upstream `network requests` rows flow through `classifyNetworkRequestFailure` / `summarizeNetworkFailures` in `network.ts` for QA analysis (`analyzeQaPresetResults` in `extensions/agent-browser/index.ts`) and for actionable-vs-benign lines plus request-specific nextActions in `network requests` presentation (`extensions/agent-browser/lib/results/presentation/diagnostics.ts`).
|
|
781
781
|
- Artifact verification: `ArtifactVerificationSummary` / `ArtifactVerificationEntry` types live in `contracts.ts`. `buildArtifactVerificationSummary`, `getArtifactVerificationEntry`, and `getManifestVerificationEntry` in `presentation/artifacts.ts` merge each resolved file artifact with manifest rows whose `storageScope` is not `explicit-path` (those rows duplicate file artifacts) and whose `path` is in the current result’s spill path set. Presentation then fails closed with `failureCategory: "artifact-missing"` when any resolved non-pending file artifact has `exists: false`; pending recording-start artifacts and non-explicit spill rows do not trigger this failure. If no hard-missing file artifact exists, successful presentation merges run `classifyPresentationSuccessCategory`, which can still force `successCategory: "artifact-unverified"` for unverified counts before delegating to `classifyAgentBrowserSuccessCategory`.
|
|
782
782
|
- Inner success categories (`classifyAgentBrowserSuccessCategory` in `categories.ts`, after verification counts are clear): if `inspection` is true → `"inspection"`; else if any non-pending artifact lacks confirmed on-disk presence (`exists !== true`) and was not upgraded to an `artifact-missing` failure → `"artifact-unverified"`; else if there is a `savedFile` or any `artifacts` → `"artifact-saved"`; else → `"completed"`.
|
|
783
783
|
- Failure: the classifier walks a single ordered chain (first match wins): `confirmation-required` → `timeout` → `missing-binary` → `parse-failure` → `aborted` → `policy-blocked` → `cleanup-failed` → `tab-drift` → `stale-ref` (including “unknown ref” text and a narrow `@eN` plus “element not found” heuristic) → `selector-unsupported` → `selector-not-found` → `download-not-verified` (download / wait-download style failures) → `validation-error` when a wrapper `validationError` is present → default `upstream-error`. Wrapper-known missing artifact checks pass an explicit `artifact-missing` category rather than relying on this text classifier.
|
|
@@ -848,11 +848,12 @@ Worth doing in v1:
|
|
|
848
848
|
- TUI display → custom `agent_browser` call/result rendering with colorized command/output text and a built-in-style collapsed view for long visible output; top-level native modes render as `agent_browser qa → batch --bail`, `agent_browser job → batch --bail` by default (`agent_browser job → batch` when `failFast:false`), or `agent_browser semanticAction → find …` so reviewers can see both the native input mode and compiled upstream command; failed results keep `resultCategory` / `failureCategory` visible before truncated output; `ctrl+o` expansion reveals the full rendered tool result without changing the model-facing content
|
|
849
849
|
- snapshots → origin + ref count + main-content-first compact preview, with the raw snapshot spill path printed directly in content and kept in `details.fullOutputPath` plus `details.artifactManifest` when the inline result would otherwise be too large
|
|
850
850
|
- oversized generic outputs such as large `eval --stdin` payloads → compact preview plus the actual spill file path instead of dumping the whole payload into model context
|
|
851
|
+
- `read [url]` → upstream `data.content` first, with source/content-type/status/final-URL metadata retained in `details.data`; explicit fetched URLs are diagnostic-only and do not replace the active browser tab target, while `read --timeout <ms>` extends the wrapper subprocess budget across upstream's per-request `.md` and ancestor-`llms.txt` fallback sequence when needed
|
|
851
852
|
- extraction-style commands like `eval --stdin` and `get title` → scalar-first text with lightweight origin context when available
|
|
852
853
|
- navigation actions like `click`, `back`, `forward`, and `reload` → lightweight post-action title/url summary when available
|
|
853
854
|
- tab lists → compact summary/table
|
|
854
855
|
- stream status → enabled/connected/port summary plus WebSocket URL and frame format when a port is known; `stream enable` errors that only say streaming is already enabled are normalized to a successful idempotent no-op with `details.data.alreadyEnabled: true` and status/disable nextActions; if the caller explicitly passed `--json`, visible text is valid JSON instead of a prose summary
|
|
855
|
-
- diagnostic/status families (`session`, `session list`, `profiles`, `doctor`, `auth list`/`show`, `cookies`, `storage`, `dialog`, `frame`, `state`, `network requests`, `console`, `errors`, and dashboard start/stop/status outputs) → compact readable summaries with counts and stable fields; `doctor` renders status/check/fix rows even when upstream puts those fields at the top level of its JSON envelope; `session list` and `tab list` keep names/labels/active markers/titles/URLs readable instead of opaque generated ids only; `network requests` and `console` previews label their scope as the upstream session aggregate unless upstream or a URL-opening QA preset explicitly cleared/filtered the buffers first; network request lists include an actionable-vs-benign failed-request summary and mark low-impact browser icon failures separately; active route mocks can add failed/pending/CORS route diagnostics; `data:image` artifact request rows are hidden from compact previews while preserved in raw details; request-detail URLs from `network request` remain diagnostic-only rather than session page targets; large log/request/error outputs use previews plus `fullOutputPath` spill files; sensitive nested auth/header/token fields are not expanded in the model-facing text
|
|
856
|
+
- diagnostic/status families (`session`, `session list`, `profiles`, `doctor`, `auth list`/`show`, `cookies`, `storage`, `dialog`, `frame`, `state`, `network requests`, `console`, `errors`, and dashboard start/stop/status outputs) → compact readable summaries with counts and stable fields; `doctor` renders status/check/fix rows even when upstream puts those fields at the top level of its JSON envelope; `session list` and `tab list` keep names/labels/active markers/titles/URLs readable instead of opaque generated ids only; `network requests` and `console` previews label their scope as the upstream session aggregate unless upstream or a URL-opening QA preset explicitly cleared/filtered the buffers first; network request lists include an actionable-vs-benign failed-request summary and mark low-impact browser icon failures separately; active route mocks can add failed/pending/CORS route diagnostics; `data:image` artifact request rows are hidden from compact previews while preserved in raw details; request-detail URLs from `network request` and fetched URLs from explicit `read <url>` remain diagnostic-only rather than session page targets; large log/request/error outputs use previews plus `fullOutputPath` spill files; sensitive nested auth/header/token fields are not expanded in the model-facing text
|
|
856
857
|
- trace/profiler owner conflicts → when the wrapper has observed one owner active for a session, block conflicting starts/stops with "wrapper believes ..." wording because upstream or external CLI use can desynchronize wrapper-local state
|
|
857
858
|
|
|
858
859
|
## Missing binary behavior
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-agent-browser-native",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.70",
|
|
4
4
|
"description": "pi extension that exposes agent-browser as a native tool for browser automation",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"author": "Mitch Fultz (https://github.com/fitchmultz)",
|
|
@@ -94,7 +94,6 @@
|
|
|
94
94
|
"prepublishOnly": "npm run verify -- release && npm pack --dry-run",
|
|
95
95
|
"build": "node ./scripts/build.mjs",
|
|
96
96
|
"startup-profile": "node ./scripts/profile-startup.mjs",
|
|
97
|
-
"prepack": "npm run build",
|
|
98
97
|
"prepare": "node ./scripts/prepare.mjs"
|
|
99
98
|
},
|
|
100
99
|
"packageManager": "npm@11.14.0",
|
|
@@ -3,15 +3,14 @@
|
|
|
3
3
|
|
|
4
4
|
import { CAPABILITY_BASELINE } from "./scripts/agent-browser-capability-baseline.mjs";
|
|
5
5
|
|
|
6
|
-
|
|
7
|
-
|
|
6
|
+
const PLATFORM_SMOKE_AGENT_BROWSER_VERSION = CAPABILITY_BASELINE.targetVersion;
|
|
7
|
+
const PLATFORM_SMOKE_UBUNTU_IMAGE = `pi-agent-browser-native-platform:node24-agent-browser${PLATFORM_SMOKE_AGENT_BROWSER_VERSION}`;
|
|
8
8
|
|
|
9
9
|
export default {
|
|
10
10
|
packageName: "pi-agent-browser-native",
|
|
11
11
|
artifactRoot: ".artifacts/platform-smoke",
|
|
12
12
|
requiredTargets: ["macos", "ubuntu", "windows-native"],
|
|
13
13
|
requiredSuites: ["platform-build", "browser-dogfood-smoke"],
|
|
14
|
-
supportedTargets: ["macos", "ubuntu", "windows-native"],
|
|
15
14
|
requiredCrabbox: {
|
|
16
15
|
install: "Homebrew package or PLATFORM_SMOKE_CRABBOX override",
|
|
17
16
|
minVersion: "0.26.0",
|
|
@@ -14,8 +14,8 @@ export const COMMAND_REFERENCE_BASELINE_BLOCK_IDS = Object.freeze(["upstream-bas
|
|
|
14
14
|
|
|
15
15
|
const sourceEvidence = Object.freeze({
|
|
16
16
|
repository: "vercel-labs/agent-browser",
|
|
17
|
-
upstreamHead: "
|
|
18
|
-
upstreamPackageVersion: "0.32.
|
|
17
|
+
upstreamHead: "6ede7a9470ac4b681cabf838af8668b9aa99e957",
|
|
18
|
+
upstreamPackageVersion: "0.32.2",
|
|
19
19
|
inspectedSources: Object.freeze([
|
|
20
20
|
"agent-browser --version",
|
|
21
21
|
"agent-browser --help",
|
|
@@ -27,11 +27,14 @@ const sourceEvidence = Object.freeze({
|
|
|
27
27
|
"agent-browser.schema.json",
|
|
28
28
|
"cli/src/commands.rs",
|
|
29
29
|
"cli/src/flags.rs",
|
|
30
|
+
"cli/src/read.rs",
|
|
30
31
|
"cli/src/doctor/webgpu.rs",
|
|
31
32
|
"cli/src/native/actions.rs",
|
|
32
33
|
"cli/src/native/daemon.rs",
|
|
33
34
|
"docs/src/app/webgpu/page.mdx",
|
|
34
35
|
"packages/@agent-browser/eve/README.md",
|
|
36
|
+
"packages/@agent-browser/eve/package.json",
|
|
37
|
+
"packages/@agent-browser/eve/test/extension.test.mjs",
|
|
35
38
|
"packages/@agent-browser/sandbox/README.md",
|
|
36
39
|
"packages/@agent-browser/sandbox/src/shared.ts",
|
|
37
40
|
"packages/@agent-browser/sandbox/src/vercel.ts",
|
|
@@ -57,6 +60,7 @@ const helpCommands = Object.freeze([
|
|
|
57
60
|
helpCommand("core skill full", ["skills", "get", "core", "--full"]),
|
|
58
61
|
helpCommand("vercel sandbox skill full", ["skills", "get", "vercel-sandbox", "--full"]),
|
|
59
62
|
helpCommand("open help", ["open", "--help"]),
|
|
63
|
+
helpCommand("read help", ["read", "--help"]),
|
|
60
64
|
helpCommand("click help", ["click", "--help"]),
|
|
61
65
|
helpCommand("key help", ["key", "--help"]),
|
|
62
66
|
helpCommand("scroll help", ["scroll", "--help"]),
|
|
@@ -154,6 +158,13 @@ const inventorySections = Object.freeze([
|
|
|
154
158
|
"open <url>",
|
|
155
159
|
"goto <url>",
|
|
156
160
|
"navigate <url>",
|
|
161
|
+
"read [url]",
|
|
162
|
+
"read <url> --raw",
|
|
163
|
+
"read <url> --require-md",
|
|
164
|
+
"read <url> --llms <index|full>",
|
|
165
|
+
"read <url> --outline",
|
|
166
|
+
"read <url> --filter <text>",
|
|
167
|
+
"read <url> --timeout <ms>",
|
|
157
168
|
"click <sel>",
|
|
158
169
|
"click <sel> --new-tab",
|
|
159
170
|
"dblclick <sel>",
|
|
@@ -229,6 +240,14 @@ const inventorySections = Object.freeze([
|
|
|
229
240
|
["open help", "open [url]"],
|
|
230
241
|
["open help", "aliases still require a URL."],
|
|
231
242
|
root("open <url>"),
|
|
243
|
+
root("read [url]"),
|
|
244
|
+
["read help", "read [url]"],
|
|
245
|
+
["read help", "--raw"],
|
|
246
|
+
["read help", "--require-md"],
|
|
247
|
+
["read help", "--llms <index|full>"],
|
|
248
|
+
["read help", "--outline"],
|
|
249
|
+
["read help", "--filter <text>"],
|
|
250
|
+
["read help", "--timeout <ms>"],
|
|
232
251
|
root("click <sel>"),
|
|
233
252
|
["click help", "--new-tab"],
|
|
234
253
|
root("dblclick <sel>"),
|
|
@@ -791,7 +810,7 @@ const inventorySections = Object.freeze([
|
|
|
791
810
|
]);
|
|
792
811
|
|
|
793
812
|
export const CAPABILITY_BASELINE = Object.freeze({
|
|
794
|
-
targetVersion: "0.32.
|
|
813
|
+
targetVersion: "0.32.2",
|
|
795
814
|
sourceEvidence,
|
|
796
815
|
helpCommands,
|
|
797
816
|
inventorySections,
|
|
@@ -126,8 +126,7 @@ export async function main(argv = process.argv.slice(2)) {
|
|
|
126
126
|
const { runTargetSuites } = await import("./platform-smoke/targets.mjs");
|
|
127
127
|
const targets = args.target ? args.target.split(",").map((name) => name.trim()).filter(Boolean) : config.requiredTargets;
|
|
128
128
|
const suites = args.suite ? [args.suite] : config.requiredSuites;
|
|
129
|
-
|
|
130
|
-
validateNames("target", targets, supportedTargets);
|
|
129
|
+
validateNames("target", targets, config.requiredTargets);
|
|
131
130
|
validateNames("suite", suites, config.requiredSuites);
|
|
132
131
|
const runs = targets.map(async (targetName) => {
|
|
133
132
|
console.log(`\n=== Target: ${targetName} ===`);
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
export * from "./browser-run/index.js";
|
|
@@ -1,20 +0,0 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* Purpose: Compatibility barrel for focused result modules.
|
|
3
|
-
* Responsibilities: Preserve the historical `./results/shared.js` import surface while delegating all logic to focused files.
|
|
4
|
-
* Scope: Re-exports only; do not add runtime policy here.
|
|
5
|
-
* Usage: Existing internal imports may keep using this path during migration, while new code should prefer focused modules.
|
|
6
|
-
* Invariants/Assumptions: This file intentionally contains no business logic so `shared` cannot grow back into a catch-all module.
|
|
7
|
-
*/
|
|
8
|
-
export * from "./contracts.js";
|
|
9
|
-
export * from "./categories.js";
|
|
10
|
-
export * from "./action-recommendations.js";
|
|
11
|
-
export * from "./artifact-manifest.js";
|
|
12
|
-
export * from "./artifact-state.js";
|
|
13
|
-
export * from "./editable-ref-evidence.js";
|
|
14
|
-
export * from "./network.js";
|
|
15
|
-
export * from "./network-routes.js";
|
|
16
|
-
export * from "./next-actions.js";
|
|
17
|
-
export * from "./recovery-actions.js";
|
|
18
|
-
export * from "./recovery-next-actions.js";
|
|
19
|
-
export * from "./selector-recovery.js";
|
|
20
|
-
export * from "./text.js";
|