pi-agent-browser-native 0.2.70 → 0.2.71
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/README.md +2 -2
- package/dist/extensions/agent-browser/lib/launch-scoped-flags.js +4 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/scroll-shims.js +42 -10
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare.js +45 -28
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output.js +2 -1
- package/dist/extensions/agent-browser/lib/playbook.js +1 -1
- package/dist/extensions/agent-browser/lib/process.js +9 -3
- package/docs/ARCHITECTURE.md +2 -1
- package/docs/COMMAND_REFERENCE.md +2 -2
- package/docs/RELEASE.md +1 -1
- package/docs/SUPPORT_MATRIX.md +16 -15
- package/docs/TOOL_CONTRACT.md +7 -6
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,23 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.71 - 2026-07-18
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
|
|
7
|
+
- Applied the same managed-session `AGENT_BROWSER_IDLE_TIMEOUT_MS` to top-level commands and every wrapper helper subprocess. With upstream 0.32.2, missing the value on hidden snapshots, tab lists, navigation summaries, and diagnostics could restart the background browser, replace the active page with `about:blank`, and make a freshly captured `@ref` fail on the next click or select.
|
|
8
|
+
- Refreshed the remaining active tab URL/title after `tab close` so subsequent snapshots, ref guards, and interactions no longer inherit the closed tab's target.
|
|
9
|
+
- Made ordinary document `scroll <direction> [amount]` deterministic before upstream wheel fallback, including pages such as Artificial Analysis whose smooth-scroll CSS previously left large scroll commands at offset zero.
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- Extended the real-upstream contract with snapshot-ref native selection and stable-id/label tab lifecycle coverage, and corrected fake/docs tab examples from unsupported positional `tab 0` to stable `tab t1`.
|
|
14
|
+
|
|
15
|
+
### Validation
|
|
16
|
+
|
|
17
|
+
- Passed `npm run verify` (590 tests passed, 2 opt-in skips), live command-reference verification, the expanded 2/2 real-upstream contract, deterministic dogfood, benchmark, packaged Pi smoke, and three-sample startup profiling (50.1 ms maximum against the 250 ms budget).
|
|
18
|
+
- Passed isolated checkout-loaded Pi dogfood on Artificial Analysis, React, GitHub, and a deterministic select/tab fixture. Artificial Analysis document scroll moved from offset 0 to 700; snapshot-ref select/click, stable-id/label tab switching, post-close target refresh, and post-close interaction completed with no background restarts, `about:blank` resets, or spurious stale-ref failures.
|
|
19
|
+
- Passed `npm run verify -- release`, including configured-source reload/relaunch lifecycle, packaged Pi smoke, and macOS/Ubuntu/native-Windows Crabbox `platform-build` plus `browser-dogfood-smoke`; all provider leases and browser sessions were cleaned.
|
|
20
|
+
|
|
3
21
|
## 0.2.70 - 2026-07-18
|
|
4
22
|
|
|
5
23
|
### Changed
|
package/README.md
CHANGED
|
@@ -74,13 +74,13 @@ The result is optimized for agent work:
|
|
|
74
74
|
| Agents build fragile shell commands | Exposes `agent_browser` with exact `args`, an optional `semanticAction` shorthand for common `find` flows and native `select`, constrained `job` / `qa` presets, experimental `sourceLookup` / `networkSourceLookup` that compile short workflows to `batch`, top-level `electron` for desktop lifecycle, plus controlled `stdin` and `sessionMode` | `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/input-modes/`, [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md) |
|
|
75
75
|
| Page snapshots are too large or viewport-blind | Shows compact, main-content-first summaries, surfaces an `Omitted high-value controls` section (plus `details.data.highValueControlRefIds`) when dense pages or desktop host screens hide editables, named surfaces/tabs, primary action buttons, and high-signal named links such as repository results from the trimmed ref lists, supports wrapper-side `snapshot -i --search <text>` / `--filter role=<role>` to trim dense pages while preserving full `details.refSnapshot`, supports `snapshot --viewport` for scroll/viewport metadata, supports `snapshot --diff` for quick ref-map deltas versus the prior tracked snapshot, and stores full raw output in spill files when needed | `extensions/agent-browser/lib/results/snapshot.ts`, `extensions/agent-browser/lib/orchestration/browser-run/prepare.ts`, `test/agent-browser.presentation.test.ts`, `test/agent-browser.extension-validation.test.ts` |
|
|
76
76
|
| Screenshots/downloads get lost in text | Normalizes artifact paths, creates missing parent directories, saves simple loopback anchor downloads to the requested path when possible, and reports existence, size, cwd, session, and repair status | [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#download-screenshot-and-pdf-files) |
|
|
77
|
-
| Profile restores and tab drift confuse agents | Tracks managed sessions, re-selects target tabs after observed drift, rehydrates branch-backed session state on Pi session-tree changes, and pins later commands only for sessions with drift/restored-session risk | generated tab-recovery notes below; `test/agent-browser.extension-tab-recovery.test.ts` (drift and about:blank recovery), `test/agent-browser.
|
|
77
|
+
| Profile restores and tab drift confuse agents | Tracks managed sessions, keeps every upstream helper probe on the same idle-timeout launch configuration so the background browser is not restarted between a snapshot and action, re-selects target tabs after observed drift, refreshes the active target after `tab close`, rehydrates branch-backed session state on Pi session-tree changes, and pins later commands only for sessions with drift/restored-session risk | generated tab-recovery notes below; `test/agent-browser.extension-tab-recovery.test.ts` (drift and about:blank recovery), `test/agent-browser.extension-tabs.test.ts` (post-close target), `test/agent-browser.extension-ref-guards.test.ts` (snapshot/action environment and session-tree rehydration), `test/agent-browser.resume-state.test.ts` (persisted session / resume planning) |
|
|
78
78
|
| Auth/profile workflows can leak secrets | Supports `auth save --password-stdin` and redacts sensitive args, URLs, stdout/stderr, details, and parse-failure spills | `test/agent-browser.extension-security-redaction.test.ts` |
|
|
79
79
|
| Stateful cookies/storage/auth output bloats or leaks context | Presentation layer redacts `details.data` for cookies and credential-like storage values while keeping low-risk local QA values such as `theme: dark` readable; recursively scrubs other structured upstream JSON (network, diff, trace/profiler, stream, dashboard, chat, auth, dialog, frame, state, and similar) using sensitive key names plus string heuristics; masks sensitive argv flags and positionals; scrubs secrets from failed batch step errors; and exposes a compact redacted `batch` matrix on top-level `details.data` | `extensions/agent-browser/lib/results/presentation.ts`, `extensions/agent-browser/lib/results/presentation/diagnostics.ts`, `extensions/agent-browser/lib/runtime.ts`, `test/agent-browser.presentation-diagnostics.test.ts` |
|
|
80
80
|
| Stale `@eN` refs fail mysteriously | Records per-session `details.refSnapshot`, rejects mismatched URLs / unknown refs / unsafe `batch` stdin ordering before spawn, adds recovery guidance to rerun `snapshot -i` or use stable `find` locators | `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/session-page-state.ts`, `test/agent-browser.session-page-state.test.ts`, `test/agent-browser.results.test.ts`, `test/agent-browser.extension-ref-guards.test.ts`, `test/agent-browser.extension-semantic-recovery.test.ts` |
|
|
81
81
|
| Agents need stable success/failure buckets | Exposes bounded `resultCategory`, `successCategory`, and `failureCategory` on tool `details` for branching without parsing prose; a `tool_result` hook also aligns real Pi `isError` semantics, naming `Pi tool isError: true` in prose output while preserving parseable caller-requested `--json` output | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/results/categories.ts`, `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/pi-tool-rendering.ts`, `test/agent-browser.results.test.ts`, `test/agent-browser.extension-validation.test.ts`, `test/agent-browser.pi-pipeline.test.ts` |
|
|
82
82
|
| Clicks can report success without the page receiving the event | Top-level non-Electron direct `click` calls on `xpath=` targets or role-gated current `@e…` refs (`button`, `checkbox`, `menuitem`, `radio`, `switch`, `tab`) install a bounded target-specific DOM-event probe; eligible `@e…` refs use the latest snapshot role/name metadata, and duplicate-name refs use snapshot-order `duplicateIndex` rather than requiring a unique name. If upstream reports success but no trusted event reaches the resolved target, the wrapper fails the tool, exposes `details.clickDispatch`, and suggests explicit retry/inspect next actions (no in-page replay), including a nested-scroll `scrollintoview` action when the probe sees the target outside a scroll container or viewport. Unresolved locator clicks such as raw `find … click` are left upstream-owned to avoid false failures for frame-scoped targets. Other click results still expose `details.pageChangeSummary`, and unchanged-URL clicks can surface evidence-backed `details.overlayBlockers` candidates. | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/orchestration/browser-run/click-dispatch.ts`, `extensions/agent-browser/lib/results/presentation/navigation.ts`, `test/agent-browser.presentation.test.ts`, `test/agent-browser.extension-click-dispatch.test.ts` |
|
|
83
|
-
| Dashboard scroll commands can look successful while nothing moves |
|
|
83
|
+
| Dashboard scroll commands can look successful while nothing moves | Handles standard `scroll <dir> [px]` against the document first (including pages whose smooth-scroll CSS defeats upstream wheel timing), falls back upstream when the document cannot move, and samples viewport/containers around the fallback; unchanged positions produce `details.scrollNoop`, visible recovery guidance, and exact snapshot/screenshot checks | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#core-page-and-element-commands), `test/agent-browser.extension-validation.test.ts` |
|
|
84
84
|
| Dropdown/combobox clicks can focus or hit native option box-model errors | Adds first-class `select <selector> <value...>` paths through raw `args`, `semanticAction`, and `job`; for custom combobox clicks, detects focused controls with explicit `aria-expanded` state but no visible options and returns `details.comboboxFocus` plus exact recovery `nextActions` | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#core-page-and-element-commands), `extensions/agent-browser/lib/input-modes/semantic-action.ts`, `test/agent-browser.extension-input-modes.test.ts`, `test/agent-browser.extension-validation.test.ts` |
|
|
85
85
|
| Recording workflows fail late when `ffmpeg` is missing | After successful `record start` / `record restart`, warns when `ffmpeg` is not on `PATH` so agents can install or fix PATH before `record stop` | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#diff-debug-and-streaming), `test/agent-browser.extension-validation.test.ts` |
|
|
86
86
|
| Direct binary help may be blocked in agent sessions | Publishes a repo-readable command reference and verifies it against the target upstream version | `npm run verify` |
|
|
@@ -36,6 +36,10 @@ export const LAUNCH_SCOPED_FLAG_DEFINITIONS = [
|
|
|
36
36
|
flag: "--init-script",
|
|
37
37
|
reason: "registers page init scripts before the upstream browser session is launched",
|
|
38
38
|
},
|
|
39
|
+
{
|
|
40
|
+
flag: "--idle-timeout",
|
|
41
|
+
reason: "configures background browser lifecycle for the launched session",
|
|
42
|
+
},
|
|
39
43
|
{
|
|
40
44
|
flag: "--device",
|
|
41
45
|
reason: "selects the provider device for the upstream launch",
|
|
@@ -53,6 +53,7 @@ function buildScrollResult(options) {
|
|
|
53
53
|
compatibilityWorkaround: options.compatibilityWorkaround,
|
|
54
54
|
data: options.result,
|
|
55
55
|
effectiveArgs: options.effectiveArgs,
|
|
56
|
+
exitCode: options.succeeded ? 0 : 1,
|
|
56
57
|
nextActions: options.succeeded ? undefined : applyNamespaceToNextActions(buildScrollNoopNextActions(options.sessionName), options.namespace),
|
|
57
58
|
[options.scrollField]: options.scrollValue,
|
|
58
59
|
sessionMode: options.sessionMode,
|
|
@@ -79,24 +80,52 @@ export async function tryContainerScroll(options) {
|
|
|
79
80
|
return buildScrollResult({ ...options, command: "scroll", message, result, scrollField: "scrollContainer", scrollValue: { request, result }, succeeded });
|
|
80
81
|
}
|
|
81
82
|
function getPageScrollToRequest(commandTokens) {
|
|
82
|
-
if (commandTokens[0] !== "scroll"
|
|
83
|
+
if (commandTokens[0] !== "scroll")
|
|
83
84
|
return undefined;
|
|
84
|
-
|
|
85
|
-
|
|
85
|
+
if (commandTokens[1]?.toLowerCase() === "to") {
|
|
86
|
+
const target = commandTokens[2]?.toLowerCase();
|
|
87
|
+
return target === "end" || target === "top" ? { target } : undefined;
|
|
88
|
+
}
|
|
89
|
+
const direction = commandTokens[1]?.toLowerCase();
|
|
90
|
+
if (!SCROLL_CONTAINER_DIRECTIONS.has(direction) || commandTokens.length > 3)
|
|
91
|
+
return undefined;
|
|
92
|
+
const amount = commandTokens[2];
|
|
93
|
+
if (amount && (!/^\d+(?:\.\d+)?(?:px|%)?$/.test(amount) || Number(amount.replace(/(?:px|%)$/, "")) <= 0))
|
|
94
|
+
return undefined;
|
|
95
|
+
return { amount, direction: direction };
|
|
86
96
|
}
|
|
87
97
|
function buildPageScrollToScript(request) {
|
|
88
98
|
return `(() => {
|
|
89
|
-
const target = ${JSON.stringify(request.target)};
|
|
99
|
+
const target = ${JSON.stringify("target" in request ? request.target : undefined)};
|
|
100
|
+
const direction = ${JSON.stringify("direction" in request ? request.direction : undefined)};
|
|
101
|
+
const amountToken = ${JSON.stringify("amount" in request ? request.amount ?? "" : "")};
|
|
102
|
+
const request = target ? { target } : { direction, amount: amountToken || undefined };
|
|
90
103
|
const scroller = document.scrollingElement || document.documentElement || document.body;
|
|
91
|
-
if (!scroller) return { status: "no-scroller",
|
|
104
|
+
if (!scroller) return { status: "no-scroller", ...request };
|
|
92
105
|
const before = { scrollLeft: scroller.scrollLeft, scrollTop: scroller.scrollTop, scrollHeight: scroller.scrollHeight, scrollWidth: scroller.scrollWidth, clientHeight: scroller.clientHeight, clientWidth: scroller.clientWidth };
|
|
93
|
-
const
|
|
94
|
-
const
|
|
95
|
-
|
|
106
|
+
const axis = direction === "left" || direction === "right" ? "x" : "y";
|
|
107
|
+
const extent = axis === "x" ? scroller.clientWidth : scroller.clientHeight;
|
|
108
|
+
const parseAmount = () => {
|
|
109
|
+
if (!amountToken) return Math.max(1, Math.floor(extent * 0.8));
|
|
110
|
+
if (amountToken.endsWith("%")) {
|
|
111
|
+
const value = Number(amountToken.slice(0, -1));
|
|
112
|
+
return Number.isFinite(value) ? Math.max(1, Math.floor(extent * value / 100)) : Math.max(1, Math.floor(extent * 0.8));
|
|
113
|
+
}
|
|
114
|
+
const pixels = Number(amountToken.replace(/px$/, ""));
|
|
115
|
+
return Number.isFinite(pixels) && pixels > 0 ? Math.floor(pixels) : Math.max(1, Math.floor(extent * 0.8));
|
|
116
|
+
};
|
|
117
|
+
const delta = parseAmount() * (direction === "up" || direction === "left" ? -1 : 1);
|
|
118
|
+
const nextTop = target === "top" ? 0 : target === "end" ? Math.max(0, scroller.scrollHeight - scroller.clientHeight) : axis === "y" ? scroller.scrollTop + delta : scroller.scrollTop;
|
|
119
|
+
const nextLeft = axis === "x" ? scroller.scrollLeft + delta : scroller.scrollLeft;
|
|
120
|
+
const priorBehavior = scroller.style.scrollBehavior;
|
|
121
|
+
scroller.style.scrollBehavior = "auto";
|
|
96
122
|
window.scrollTo(nextLeft, nextTop);
|
|
123
|
+
scroller.scrollLeft = nextLeft;
|
|
124
|
+
scroller.scrollTop = nextTop;
|
|
97
125
|
const after = { scrollLeft: scroller.scrollLeft, scrollTop: scroller.scrollTop, scrollHeight: scroller.scrollHeight, scrollWidth: scroller.scrollWidth, clientHeight: scroller.clientHeight, clientWidth: scroller.clientWidth };
|
|
126
|
+
scroller.style.scrollBehavior = priorBehavior;
|
|
98
127
|
const moved = before.scrollLeft !== after.scrollLeft || before.scrollTop !== after.scrollTop;
|
|
99
|
-
return { status: moved ? "scrolled" : "no-movement",
|
|
128
|
+
return { status: moved ? "scrolled" : "no-movement", ...request, before, after };
|
|
100
129
|
})()`;
|
|
101
130
|
}
|
|
102
131
|
export async function tryPageScrollTo(options) {
|
|
@@ -108,6 +137,9 @@ export async function tryPageScrollTo(options) {
|
|
|
108
137
|
if (!isRecord(result) || typeof result.status !== "string")
|
|
109
138
|
return undefined;
|
|
110
139
|
const succeeded = result.status === "scrolled";
|
|
111
|
-
|
|
140
|
+
if (!succeeded && "direction" in request)
|
|
141
|
+
return undefined;
|
|
142
|
+
const description = "target" in request ? `to ${request.target}` : `${request.direction}${request.amount ? ` by ${request.amount}` : ""}`;
|
|
143
|
+
const message = succeeded ? `Scrolled page ${description}.` : `Scroll ${description} completed with no observed movement (${result.status}).`;
|
|
112
144
|
return buildScrollResult({ ...options, command: "scroll", message, result, scrollField: "scrollPage", scrollValue: { request, result }, succeeded });
|
|
113
145
|
}
|
|
@@ -253,6 +253,18 @@ async function collectSamePageRefFreshnessPreflight(options) {
|
|
|
253
253
|
return undefined;
|
|
254
254
|
return { message: mismatch.message, refIds: mismatch.refIds, snapshot: snapshotWithTarget };
|
|
255
255
|
}
|
|
256
|
+
function getIdleTimeoutMismatch(args, configuredValue) {
|
|
257
|
+
for (let index = 0; index < args.length; index += 1) {
|
|
258
|
+
const token = args[index];
|
|
259
|
+
if (token !== "--idle-timeout" && !token.startsWith("--idle-timeout="))
|
|
260
|
+
continue;
|
|
261
|
+
const requestedToken = token.includes("=") ? token.slice(token.indexOf("=") + 1) : args[++index];
|
|
262
|
+
if (!requestedToken || !/^\d+$/.test(requestedToken) || Number(requestedToken) === Number(configuredValue))
|
|
263
|
+
continue;
|
|
264
|
+
return `--idle-timeout ${requestedToken} conflicts with this Pi process's managed-session idle timeout (${configuredValue} ms). Restart Pi with PI_AGENT_BROWSER_IMPLICIT_SESSION_IDLE_TIMEOUT_MS=${requestedToken} and omit --idle-timeout; changing the launch value for one call can restart the upstream browser and discard the active tab.`;
|
|
265
|
+
}
|
|
266
|
+
return undefined;
|
|
267
|
+
}
|
|
256
268
|
function isPasswordStdinAuthSave(options) {
|
|
257
269
|
return options.command === "auth" && options.commandTokens[1] === "save" && options.commandTokens.includes("--password-stdin");
|
|
258
270
|
}
|
|
@@ -340,6 +352,9 @@ export async function prepareBrowserRun(options) {
|
|
|
340
352
|
managedSessionNamespace: state.managedSessionNamespace,
|
|
341
353
|
sessionMode,
|
|
342
354
|
});
|
|
355
|
+
const idleTimeoutMismatch = getIdleTimeoutMismatch(preparedArgs.args, options.implicitSessionIdleTimeoutMs);
|
|
356
|
+
if (idleTimeoutMismatch)
|
|
357
|
+
executionPlan = { ...executionPlan, recoveryHint: undefined, validationError: idleTimeoutMismatch };
|
|
343
358
|
const sessionStateKey = getSessionContextKey(executionPlan.sessionName, executionPlan.namespace);
|
|
344
359
|
const priorSessionPageState = sessionPageState.get(sessionStateKey);
|
|
345
360
|
const priorSessionTabTarget = priorSessionPageState.tabTarget;
|
|
@@ -581,34 +596,36 @@ export async function prepareBrowserRun(options) {
|
|
|
581
596
|
});
|
|
582
597
|
if (networkRequestsPageFilter)
|
|
583
598
|
return { kind: "early-result", statePatch, result: networkRequestsPageFilter };
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
599
|
+
if (executionPlan.startupScopedFlags.length === 0) {
|
|
600
|
+
const containerScroll = await tryContainerScroll({
|
|
601
|
+
commandTokens,
|
|
602
|
+
compatibilityWorkaround,
|
|
603
|
+
cwd,
|
|
604
|
+
effectiveArgs: redactedEffectiveArgs,
|
|
605
|
+
redactedArgs,
|
|
606
|
+
sessionMode,
|
|
607
|
+
namespace: executionPlan.namespace,
|
|
608
|
+
sessionName: executionPlan.sessionName,
|
|
609
|
+
signal,
|
|
610
|
+
usedImplicitSession: executionPlan.usedImplicitSession,
|
|
611
|
+
});
|
|
612
|
+
if (containerScroll)
|
|
613
|
+
return { kind: "early-result", statePatch, result: containerScroll };
|
|
614
|
+
const pageScrollTo = await tryPageScrollTo({
|
|
615
|
+
commandTokens,
|
|
616
|
+
compatibilityWorkaround,
|
|
617
|
+
cwd,
|
|
618
|
+
effectiveArgs: redactedEffectiveArgs,
|
|
619
|
+
redactedArgs,
|
|
620
|
+
sessionMode,
|
|
621
|
+
namespace: executionPlan.namespace,
|
|
622
|
+
sessionName: executionPlan.sessionName,
|
|
623
|
+
signal,
|
|
624
|
+
usedImplicitSession: executionPlan.usedImplicitSession,
|
|
625
|
+
});
|
|
626
|
+
if (pageScrollTo)
|
|
627
|
+
return { kind: "early-result", statePatch, result: pageScrollTo };
|
|
628
|
+
}
|
|
612
629
|
const directAnchorDownload = await tryDirectAnchorDownload({
|
|
613
630
|
artifactManifest: state.artifactManifest,
|
|
614
631
|
commandTokens,
|
|
@@ -188,7 +188,8 @@ export async function processBrowserOutput(input) {
|
|
|
188
188
|
!navigationSummary &&
|
|
189
189
|
(shouldCaptureNavigationSummary(prepared.executionPlan.commandInfo.command, presentationEnvelope?.data) ||
|
|
190
190
|
shouldCaptureSemanticActionNavigationSummary(prepared.compiledSemanticAction, presentationEnvelope?.data) ||
|
|
191
|
-
shouldCaptureAllowedDomainNavigationSummary
|
|
191
|
+
shouldCaptureAllowedDomainNavigationSummary ||
|
|
192
|
+
(prepared.executionPlan.commandInfo.command === "tab" && prepared.executionPlan.commandInfo.subcommand === "close"))) {
|
|
192
193
|
navigationSummary = await collectNavigationSummary({ cwd, namespace: prepared.executionPlan.namespace, sessionName: prepared.executionPlan.sessionName, signal });
|
|
193
194
|
}
|
|
194
195
|
if (navigationSummary && presentationEnvelope && !Array.isArray(presentationEnvelope.data))
|
|
@@ -20,7 +20,7 @@ export const QUICK_START_GUIDELINES = [
|
|
|
20
20
|
"Locator-first clicks/fills and native select changes without hand-building argv: { semanticAction: { action: \"click\", locator: \"text\", value: \"Close\" } }, { semanticAction: { action: \"fill\", locator: \"label\", value: \"Email\", text: \"user@example.com\" } }, direct current targets such as { semanticAction: { action: \"fill\", selector: \"@e1\", text: \"prompt\" } }, or { semanticAction: { action: \"select\", selector: \"#flavor\", value: \"chocolate\" } }; add semanticAction.session when targeting a named upstream browser session; details.compiledSemanticAction shows the semantic target, while details.effectiveArgs may show a resolved current @ref for active-session role/name click/check/fill actions to avoid hidden duplicate matches; semanticAction does not expose uncheck while upstream find ... uncheck is not runtime-supported, so use raw uncheck with a stable selector or current ref; selector-not-found failures may append bounded click try-*-candidate next actions or, for fill misses with current editable refs, details.richInputRecovery with focus/click actions that do not copy fill text; stale-ref failures can return retry-semantic-action-after-stale-ref for compiled find actions when retry safety is provable.",
|
|
21
21
|
`Common advanced calls: { args: ["batch"], stdin: "[[\"open\",\"https://example.com\"],[\"snapshot\",\"-i\"]]" }, { job: { steps: [{ action: "open", url: "https://example.com" }, { action: "assertText", text: "Example Domain" }, { action: "screenshot", path: ".dogfood/example.png" }] } }, { qa: { url: "https://example.com", expectedText: "Example Domain", screenshotPath: ".dogfood/qa-example.png" } } (example.com smoke only; elsewhere match exact visible text from snapshot -i), { electron: { action: "list", query: "code" } }, { electron: { action: "launch", appName: "Visual Studio Code", handoff: "snapshot" } }, { electron: { action: "probe" } }, { qa: { attached: true, expectedText: "Explorer" } }, { args: ["eval", "--stdin"], stdin: "document.title", outputPath: "logs/page-title.json" }, { args: ["auth", "save", "name", "--password-stdin"], stdin: "<password from user-approved secret source>" }, { args: ["--profile", "Default", "open", "https://example.com/account"], sessionMode: "fresh" }, and { args: ["open", "--enable", "react-devtools", "https://example.com"], sessionMode: "fresh" }. For app pages with a native dropdown, job steps can include { action: "select", selector: "#flavor", value: "chocolate" } before the dependent assertion; for locator-friendly pages, job click/fill steps can use semantic locator fields such as { action: "fill", locator: "role", role: "searchbox", name: "Search", text: "agent browser" }; for human-paced input, job type steps can use { action: "type", selector: "#prompt", text: "hello", delayMs: 20, press: "Enter" }; delayed typing is capped at 200 characters per step, and generated per-character rows are compacted in visible batch prose while full rows remain in details.batchSteps.`,
|
|
22
22
|
"Constrained job navigation is explicit only: click (and select/submit flows that may navigate) does not prove the next page loaded; add assertUrl and/or assertText after navigation-prone steps before screenshot or later interactions. Keep jobs short around navigation, click, and rerender boundaries on dynamic React/product apps; avoid a whole checkout in one job. If a long job times out and details.timeoutPartialProgress shows a mutating incomplete step, inspect current page state and continue with a shorter job or single action instead of blindly retrying the mutating step. Example: { job: { steps: [{ action: \"open\", url: \"https://shop.example/checkout\" }, { action: \"fill\", selector: \"#email\", text: \"user@example.com\" }, { action: \"click\", selector: \"#continue\" }, { action: \"assertUrl\", url: \"**/shipping\" }, { action: \"assertText\", text: \"Shipping address\" }, { action: \"screenshot\", path: \".dogfood/shipping.png\" }] } }. Top-level click may add pageChangeSummary hints, but job never auto-inserts post-click asserts.",
|
|
23
|
-
"High-value command reference: click <selector> --new-tab opens link-like targets in a new tab; select <selector> <value...> changes native dropdown values; scroll <dir> [px]
|
|
23
|
+
"High-value command reference: click <selector> --new-tab opens link-like targets in a new tab; select <selector> <value...> changes native dropdown values; wrapper-handled scroll <dir> [px|percent] and scroll to end/top target document scrolling before upstream fallback, while scroll <selector> <dir> [px|percent] targets nested scrollers; download <selector> <path> saves a file triggered by a click; read [url] returns agent-readable text (explicit URLs prefer markdown without launching Chrome; omit the URL for rendered active-tab DOM); get title/url need no selector; get text/html/value/count <selector> and get attr <selector> <name> read elements/page state (use body for whole-page text/html); screenshot [selector] [path] captures a page or element image; pdf <path> saves a PDF; tab list and tab <tab-id-or-label> inspect or recover the active tab; react tree, react inspect <fiberId>, react renders start/stop, and react suspense introspect React after --enable react-devtools; vitals [url] measures Core Web Vitals; pushstate <url> performs SPA navigation; tap <selector> and swipe <direction> [distance] support iOS/provider touch flows.",
|
|
24
24
|
"For artifact-producing commands, read the visible artifact block and details.artifactVerification before using files: check requested path, absolute path, existence, size bytes, artifact kind, optional mediaType, status, optional limitation, and verified/missing/pending/unverified counts. details.artifacts contains per-file metadata; record start rows are pending/openRecording until record stop writes the target. The wrapper creates parent directories for direct artifact paths and can save simple loopback HTTP(S) anchor downloads directly to the requested path before upstream download fallback. Browser close does not delete explicit saved files; if close reports details.artifactCleanup, use host file tools to remove paths listed in explicitArtifactPaths (when non-empty) after inspection. If close fails with details.promptGuard.reason=requested-artifacts-missing-before-close, save the exact required artifact path before closing. For annotated screenshots inside batch, put --annotate in top-level args (for example { args: [\"--annotate\", \"batch\"], stdin: \"[[\\\"screenshot\\\",\\\"/tmp/page.png\\\"]]\" }) rather than inside the screenshot step; if annotation labels crowd a dense page, use a scoped or non-annotated screenshot plus snapshot refs instead.",
|
|
25
25
|
"When details.nextActions is present, prefer those exact native agent_browser follow-up payloads over prose guidance; they may include args, stdin, sessionMode, networkSourceLookup, safety notes, or artifactPath for saved files.",
|
|
26
26
|
];
|
|
@@ -9,6 +9,7 @@ import { spawn } from "node:child_process";
|
|
|
9
9
|
import { chmod, mkdir } from "node:fs/promises";
|
|
10
10
|
import { env as processEnv, platform as processPlatform } from "node:process";
|
|
11
11
|
import { GLOBAL_BOOLEAN_FLAGS_WITH_OPTIONAL_VALUES, GLOBAL_VALUE_FLAGS, getFlagName } from "./argv-grammar.js";
|
|
12
|
+
import { getImplicitSessionIdleTimeoutMs } from "./runtime.js";
|
|
12
13
|
import { openSecureTempFile, writeSecureTempChunk } from "./temp.js";
|
|
13
14
|
const MAX_BUFFERED_STDOUT_BYTES = 512 * 1_024;
|
|
14
15
|
const MAX_BUFFERED_STDERR_CHARS = 32_000;
|
|
@@ -16,6 +17,7 @@ const MAX_BUFFERED_STDOUT_TAIL_CHARS = 32_000;
|
|
|
16
17
|
const PROCESS_STDOUT_SPILL_FILE_PREFIX = "process-stdout";
|
|
17
18
|
const AGENT_BROWSER_SOCKET_DIR_ENV = "AGENT_BROWSER_SOCKET_DIR";
|
|
18
19
|
const AGENT_BROWSER_DEFAULT_TIMEOUT_ENV = "AGENT_BROWSER_DEFAULT_TIMEOUT";
|
|
20
|
+
const AGENT_BROWSER_IDLE_TIMEOUT_ENV = "AGENT_BROWSER_IDLE_TIMEOUT_MS";
|
|
19
21
|
const PI_AGENT_BROWSER_PROCESS_TIMEOUT_ENV = "PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS";
|
|
20
22
|
const DEFAULT_AGENT_BROWSER_SOCKET_DIR_PREFIX = "/tmp/piab";
|
|
21
23
|
export const SAFE_AGENT_BROWSER_OPERATION_TIMEOUT_MS = 25_000;
|
|
@@ -184,11 +186,15 @@ export function buildAgentBrowserProcessEnv(baseEnv = processEnv, overrides = un
|
|
|
184
186
|
export async function runAgentBrowserProcess(options) {
|
|
185
187
|
const { args, cwd, env, signal, stdin } = options;
|
|
186
188
|
const timeoutMs = options.timeoutMs ?? getAgentBrowserProcessTimeoutMs();
|
|
187
|
-
const
|
|
188
|
-
|
|
189
|
+
const processOverrides = {
|
|
190
|
+
[AGENT_BROWSER_IDLE_TIMEOUT_ENV]: String(getImplicitSessionIdleTimeoutMs()),
|
|
191
|
+
...env,
|
|
192
|
+
};
|
|
193
|
+
const explicitSocketDir = processOverrides[AGENT_BROWSER_SOCKET_DIR_ENV];
|
|
194
|
+
let effectiveEnv = explicitSocketDir === undefined ? { ...processOverrides, [AGENT_BROWSER_SOCKET_DIR_ENV]: undefined } : processOverrides;
|
|
189
195
|
const requestedSocketDir = explicitSocketDir ?? getAgentBrowserSocketDir();
|
|
190
196
|
if (requestedSocketDir && (await ensureAgentBrowserSocketDir(requestedSocketDir))) {
|
|
191
|
-
effectiveEnv = { ...
|
|
197
|
+
effectiveEnv = { ...effectiveEnv, [AGENT_BROWSER_SOCKET_DIR_ENV]: requestedSocketDir };
|
|
192
198
|
}
|
|
193
199
|
return await new Promise((resolve) => {
|
|
194
200
|
let aborted = false;
|
package/docs/ARCHITECTURE.md
CHANGED
|
@@ -139,7 +139,7 @@ V1 ownership rule:
|
|
|
139
139
|
Practical policy:
|
|
140
140
|
- preserve the current branch-visible extension-managed session across `/reload`, exact-session relaunch, `/resume`, and Pi 0.79 `session_tree` branch transitions so persisted sessions can keep following the live browser after lifecycle changes
|
|
141
141
|
- close the active extension-managed session when the originating `pi` process quits, while leaving explicit caller-provided sessions alone
|
|
142
|
-
- set an idle timeout on extension-managed sessions as a backstop for abnormal exits or cleanup failures
|
|
142
|
+
- set an idle timeout on extension-managed sessions as a backstop for abnormal exits or cleanup failures, and apply that same `AGENT_BROWSER_IDLE_TIMEOUT_MS` value to every upstream subprocess (including wrapper helper snapshots, tab lists, and navigation-summary reads) because changing the launch environment between calls can make upstream restart the background browser, discard the active tab, and invalidate fresh refs
|
|
143
143
|
- clean up process-private temp spill artifacts on shutdown, but keep persisted-session snapshot spill files in a private session-scoped artifact directory with a bounded per-session budget so `details.fullOutputPath` stays usable after reload/resume without unbounded growth
|
|
144
144
|
- keep explicit screenshots, downloads, PDFs, traces, HAR captures, and recordings written to caller-chosen paths on disk after a successful upstream close command (`close`, `quit`, or `exit`); before artifact-producing commands run, create missing parent directories for requested host paths, and for simple loopback HTML anchor downloads with resolvable HTTP(S) hrefs the wrapper may save directly to the requested path before upstream fallback. When the bounded `details.artifactManifest` has entries, successful close commands also surface `details.artifactCleanup` and a compact `Artifact lifecycle` note pointing to structured explicit paths so operators remove files with normal host tools—the native tool does not delete arbitrary user paths (`extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts`, `getArtifactCleanupGuidance`); contract in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), checklist `RQ-0079` in [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md)
|
|
145
145
|
- reconstruct the current branch-visible extension-managed session, page-scoped refs, artifact manifest, and Electron launch records from the active transcript branch on `session_start` and `session_tree` so later default calls keep following the active managed browser after resume/reload or branch switching; restore also honors successful explicit `--session <wrapper-owned> close` rows and `electron.cleanup` managed-session steps so closed wrapper-owned sessions are not resurrected
|
|
@@ -152,6 +152,7 @@ Practical policy:
|
|
|
152
152
|
- after profiled `open` / `goto` / `navigate` calls, verify the active tab still matches the returned page URL and best-effort switch back when restored profile tabs steal focus
|
|
153
153
|
- once the wrapper observes tab-drift risk for a session (profile restore correction, overlapping stale opens, or restored session state), later active-tab commands may synthesize a tiny upstream `batch` that re-selects that tab and then runs the requested command in the same upstream invocation; routine same-session commands avoid `tab list` preflights to reduce probes that can perturb upstream click behavior
|
|
154
154
|
- for sessions with observed tab-drift risk, after a successful command on a known tab target, the wrapper may best-effort restore that same target again if restored/background tabs steal focus after the command returns; routine same-session commands skip this post-command `tab list` probe
|
|
155
|
+
- after successful `tab close`, read the now-active URL/title before updating per-session page state because upstream's close payload identifies the closed tab but does not report the remaining active tab; this keeps subsequent ref guards and interactions aligned without requiring a recovery snapshot solely to repair wrapper state
|
|
155
156
|
- keep a per-session `refSnapshot` aligned with the last successful `snapshot` (including refs merged from a successful `batch` by taking the last successful `snapshot` step in batch result order): restore it from persisted tool `details` when reloading, resuming, or moving to a different Pi session-tree branch, store bounded ref role/name metadata from the same snapshot for wrapper-side current-ref diagnostics, drop it on successful close commands (`close`, `quit`, or `exit`), and refuse mutation-prone `@e…` argv before spawn when the active tab URL no longer matches the snapshot URL, when a ref id was never in that snapshot, or when `batch` stdin would reuse `@e…` on a guarded step after an earlier invalidating step without a later `snapshot` step in the same stdin array. Same-snapshot `fill @e…` rows are guarded but do not themselves set that invalidation latch, so ordinary form fills can precede a click/submit row in one batch—see [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details) for the agent-visible contract and failure text; typed per-session tab/ref/pinning state lives in `extensions/agent-browser/lib/session-page-state.ts` and is updated from `extensions/agent-browser/index.ts` after each tool result
|
|
156
157
|
- for top-level non-Electron direct `click` commands with an eligible target, install a bounded in-page target-specific event probe before upstream runs; if upstream reports success but no trusted pointer/mouse/click event reached the resolved target, fail the tool and report `details.clickDispatch` with explicit retry/inspect next actions (the wrapper does not replay clicks in-page). The probe covers `xpath=` targets and current `@e…` / `ref=` refs whose latest stored `refSnapshot.refs` role is `button`, `checkbox`, `menuitem`, `radio`, `switch`, or `tab`; it uses that role/name metadata, including snapshot-order `duplicateIndex` for duplicate-name refs, instead of taking a fresh pre-click snapshot that could recycle upstream refs. The probe is intentionally skipped for CSS selector clicks, unresolved `find … click` locators, and `batch`/`job`/`qa` click steps
|
|
157
158
|
- derive narrow prompt guards only for concrete evidence invariants: exact required screenshot paths block browser close until the artifact manifest verifies those paths. The wrapper intentionally does not infer broad business/user intent from prompt text such as order/payment/post boundaries; agents must follow those instructions themselves. The artifact guard is bounded preflight policy (`details.promptGuard`, `failureCategory: "policy-blocked"`), not a reusable browser recipe layer
|
|
@@ -610,7 +610,7 @@ Skill-source debugging note: upstream honors `AGENT_BROWSER_SKILLS_DIR` as an ov
|
|
|
610
610
|
| `tap <selector>` | Touch-oriented tap alias for iOS/provider workflows. |
|
|
611
611
|
| `swipe <direction> [distance]` | Touch-oriented swipe for iOS/provider workflows. |
|
|
612
612
|
|
|
613
|
-
On dashboards and other apps with nested scroll containers, `scroll <dir> [px]`
|
|
613
|
+
On dashboards and other apps with nested scroll containers, `scroll <dir> [px]` can miss because a page-level wheel does not move the document or the intended pane. Without startup-scoped launch flags, the wrapper first applies ordinary `scroll <up|down|left|right> [px|percent]` directly to `document.scrollingElement` with smooth scrolling temporarily disabled; successful movement reports `details.scrollPage`. If the document cannot move, it falls back to upstream wheel behavior. For large fallback calls on an existing or fresh managed session, the wrapper samples viewport and prominent scroll-container positions before and after the command; when nothing changes it prepends `Scroll completed with no observed movement`, appends `Scroll diagnostic: no observed scroll movement`, exposes `details.scrollNoop`, marks `details.data.scrolled: false`, and adds exact `details.nextActions` for a fresh `snapshot -i` and screenshot. Explicit CSS-container calls `scroll <selector> <up|down|left|right> [px|percent]` remain wrapper-handled and report `details.scrollContainer`; `scroll to end` / `scroll to top` report `details.scrollPage`. Calls with startup-scoped flags skip all helper shims so the requested launch configuration runs first. Use these paths before repeating page scrolls; when you need a specific element, prefer `scrollintoview <@ref>` or target the actual scrollable region.
|
|
614
614
|
|
|
615
615
|
Comboboxes vary by app. For native `<select>` controls, prefer raw `select <selector> <value...>`, `semanticAction: { action: "select", selector, value|values }`, or a `job` `select` step instead of clicking option refs; native option refs can be non-boxed in CDP and fail before a real selection. A `click` or `semanticAction` role/name click may focus a searchable custom combobox without opening its option list. For explicit combobox-targeted actions such as `semanticAction` role `combobox`, the wrapper checks whether a combobox-like element is focused, has explicit `aria-expanded` state, and has no visible listbox/options open; this still applies when the semantic action first resolves to a current visible `@ref` before execution. When that happens it appends `Combobox diagnostic: focused combobox did not expose visible options`, exposes `details.comboboxFocus`, and adds exact `details.nextActions` for a fresh `snapshot -i`, `press ArrowDown`, and `press Enter`. Use those instead of assuming click alone expanded the control; reserve visible option refs for custom comboboxes after a fresh snapshot shows the intended option.
|
|
616
616
|
|
|
@@ -890,7 +890,7 @@ Browser default config is conservative: it adds agent guidance for signed-in/acc
|
|
|
890
890
|
- `--download-path <path>`: default browser download directory. Environment: `AGENT_BROWSER_DOWNLOAD_PATH`.
|
|
891
891
|
- `--engine <name>`: browser engine, `chrome` by default or `lightpanda`. Environment: `AGENT_BROWSER_ENGINE`.
|
|
892
892
|
- `--no-auto-dialog`: disable automatic dismissal of alert/beforeunload dialogs. Environment: `AGENT_BROWSER_NO_AUTO_DIALOG`.
|
|
893
|
-
- `--idle-timeout <ms>`:
|
|
893
|
+
- `--idle-timeout <ms>`: launch-scoped background browser lifecycle setting. The wrapper already sets one stable `AGENT_BROWSER_IDLE_TIMEOUT_MS` for top-level and helper subprocesses. A per-call value must equal that configured value; otherwise the tool rejects it before launch and tells you to restart Pi with `PI_AGENT_BROWSER_IMPLICIT_SESSION_IDLE_TIMEOUT_MS=<ms>`. This prevents upstream from restarting the browser and discarding tabs/refs when later helper calls use a different launch environment.
|
|
894
894
|
|
|
895
895
|
### Output, provider, policy, and AI flags
|
|
896
896
|
|
package/docs/RELEASE.md
CHANGED
|
@@ -281,7 +281,7 @@ This suite requires the installed `agent-browser --version` to exactly match `sc
|
|
|
281
281
|
- **Failure shape:** `react tree` on a page opened with `--enable react-devtools` but without a React app (expects a clear missing-renderer error with session-bound `details`).
|
|
282
282
|
- **Async download:** `open` on the `/download` fixture, anchor-triggered export, then `wait --download <path>` metadata and wrapper artifact reporting for the requested path.
|
|
283
283
|
|
|
284
|
-
The default unit suite also runs `agentBrowserExtension passes through core command coverage fallback matrix` in [`test/agent-browser.extension-validation.test.ts`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/test/agent-browser.extension-validation.test.ts): a fake upstream records argv so `connect 9222`, `download` with a selector and path, `get url`, `snapshot --compact`, and `tab new` / `tab
|
|
284
|
+
The default unit suite also runs `agentBrowserExtension passes through core command coverage fallback matrix` in [`test/agent-browser.extension-passthrough-validation.test.ts`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/test/agent-browser.extension-passthrough-validation.test.ts): a fake upstream records argv so `connect 9222`, `download` with a selector and path, `get url`, `snapshot --compact`, and `tab new` / `tab t1` / `tab close` still prove `--json` plus implicit `--session` ordering without a browser. A second fake-upstream matrix in that file (`agentBrowserExtension passes through non-core network debug diff stream dashboard and chat families`) pins representative `network`, `diff`, `trace` / `profiler` / `record`, `console` / `errors` / `highlight` / `inspect` / `clipboard`, `stream`, `dashboard`, and `chat` JSON shapes plus redacted `details.data` and argv echoes without a browser. A third matrix (`agentBrowserExtension passes through provider and specialized skill workflows`) asserts provider `open` argv shapes still receive `--json` plus implicit `--session` while read-only `skills get …` stays stateless (no managed session fields) and provider credential env vars are forwarded into the fake upstream log. Extend those matrices when adding passthrough coverage that should stay out of the slow real-upstream loop.
|
|
285
285
|
|
|
286
286
|
### Real upstream suite mechanics, isolation, and troubleshooting
|
|
287
287
|
|
package/docs/SUPPORT_MATRIX.md
CHANGED
|
@@ -28,7 +28,7 @@ When upstream ships a new `agent-browser` or the inventory changes:
|
|
|
28
28
|
|
|
29
29
|
- Target upstream: `agent-browser 0.32.2` (must match `CAPABILITY_BASELINE.targetVersion` in [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs)).
|
|
30
30
|
- Source of truth: `CAPABILITY_BASELINE.inventorySections` in the same file (stable `id` keys: `skills`, `core-commands`, `state-tabs-frames-dialogs`, `network-storage-artifacts-diagnostics`, `batch-auth-setup-ai`, `options-and-env`).
|
|
31
|
-
- Status: source and wrapper adaptation are complete for the 2026-07-
|
|
31
|
+
- Status: source and wrapper adaptation are complete for the 2026-07-18 0.32.2 audit; the 0.32.1–0.32.2 changes stay isolated to the separate `@agent-browser/eve` package, while the wrapper now also keeps hidden helper subprocesses on the same idle-timeout launch configuration, refreshes the remaining active target after `tab close`, and performs deterministic document-direction scrolls before wheel fallback.
|
|
32
32
|
- High-priority support gaps: 2026-05-26 audit found sessionless local commands and command-scoped value flags needed sharper wrapper handling; runtime/tests/docs now cover those paths. The 0.28.0 rebaseline added local `mcp` and `plugin` surfaces plus plugin-backed credential login; wrapper docs/tests mark `mcp` and known `plugin` commands sessionless, with no compatibility shim for older upstream releases. The 0.29.1 rebaseline added upstream `@agent-browser/sandbox` helper-package guidance and stricter `install --with-deps` failure semantics; no new wrapper runtime mode or bundled dependency was required. The 0.30.1 rebaseline fixed upstream `wait --url` glob matching, so constrained `job.assertUrl` delegates glob and exact patterns directly to `wait --url`. The 0.31.0 rebaseline adds restore workflow and namespace/session lifecycle surfaces (`--restore`, restore checks, `--namespace`, `session id`, and `session info`) without adding a wrapper compatibility layer. The 0.31.1 rebaseline fixes upstream React renderer selection for `react tree`, `react inspect`, and `react suspense`; no wrapper runtime change was required. The 0.31.2 rebaseline adds the local-launch-only `--webgpu` preset and `doctor --webgpu`, plus periodic restore-state autosaves controlled by `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS`; the wrapper treats WebGPU as launch-scoped and leaves autosave ownership upstream. The 0.32.0 rebaseline hardens request/worker/popup/WebRTC containment behind `--allowed-domains`, fixes waits against already-complete documents, and adds a separate `@agent-browser/eve` package; the wrapper makes containment launch-scoped, retains its final-URL check as defense in depth, and adds no Eve-specific mode. The 0.32.1–0.32.2 rebaseline updates only eve compatibility/packaging, but the full current-surface audit also found the local reference had omitted upstream `read [url]` since 0.30.0; baseline/help sampling, content-first presentation, tab-target preservation, timeout budgeting, docs, and tests now cover it. Remaining upstream-owned caveat: current help still mentions `wait <selector> --state hidden` / `detached` and `find ... uncheck`, but runtime probes show those advertised shapes still fail, so wrapper docs keep `wait --fn` predicates and direct `uncheck` passthrough guidance.
|
|
33
33
|
- Post-`v0.2.29` review state: commits `eb55320` through `86abbfb` add browser guidance/smoke coverage plus `RQ-0086` click-probe reduction, `RQ-0087` same-snapshot form fill batching, `RQ-0088` current-ref fallback on locator misses, `RQ-0089` direct-upstream click mutation investigation, and `RQ-0090` stop-boundary/artifact-path guidance. Verification gates below were rerun on 2026-05-18 after those tasks landed. Constrained `job` (`RQ-0064`), the lightweight `qa` preset (`RQ-0065`), the experimental `sourceLookup` helper (`RQ-0066`), the experimental `networkSourceLookup` helper (`RQ-0067`), optional Exa/Brave-backed `agent_browser_web_search` with Pi-scoped package config (`RQ-0121`), and agent recovery for search/profile configuration failures (`RQ-0122`) are implemented; see [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#job), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#qa), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#sourcelookup), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#networksourcelookup), and [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#optional-companion-web-search). Reusable browser recipes (`RQ-0068`) are intentionally not adopted as a runtime surface; see [`ARCHITECTURE.md`](ARCHITECTURE.md#no-reusable-recipe-layer-yet).
|
|
34
34
|
|
|
@@ -51,25 +51,26 @@ Current summary:
|
|
|
51
51
|
| RQ-0133 | Upstream `agent-browser 0.31.2` rebaseline adds WebGPU launch/doctor/MCP surfaces and periodic restore-state autosaves; wrapper launch parsing/policy and docs now cover both. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0312-rebaseline) |
|
|
52
52
|
| RQ-0134 | Upstream `agent-browser 0.32.0` rebaseline hardens domain containment, fixes completed-page waits, and adds a separate eve extension package; wrapper launch policy and docs cover the relevant behavior. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0320-rebaseline) |
|
|
53
53
|
| RQ-0135 | Upstream `agent-browser 0.32.2` rebaseline updates eve packaging only and closes the wrapper's prior missing `read [url]` inventory/presentation contract. | [`docs/COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#upstream-0322-rebaseline) |
|
|
54
|
+
| RQ-0136 | Wrapper regression fix keeps helper subprocess launch environments stable, refreshes active tab state after `tab close`, and makes ordinary document-direction scrolling deterministic before upstream fallback. | [`docs/ARCHITECTURE.md`](ARCHITECTURE.md), [`docs/TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details) |
|
|
54
55
|
|
|
55
56
|
## Verification evidence
|
|
56
57
|
|
|
57
|
-
Re-run the gates below before each release; this table records what the closure audit exercised. The 0.32.2 / Pi 0.80.10 local rebaseline gates passed on 2026-07-
|
|
58
|
+
Re-run the gates below before each release; this table records what the closure audit exercised. The 0.32.2 / Pi 0.80.10 local rebaseline gates passed on 2026-07-18; the prior 0.32.0 / Pi 0.80.9 release-composition and platform rows remain explicitly historical evidence.
|
|
58
59
|
|
|
59
60
|
| Gate | Evidence | Status |
|
|
60
61
|
| --- | --- | --- |
|
|
61
|
-
| Default local gate | `npm run verify` checks generated playbook drift, clean-builds generated `dist/`, runs `tsc --noEmit`, unit/fake tests, generated command-reference blocks, and live command-reference sampling. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-
|
|
62
|
+
| Default local gate | `npm run verify` checks generated playbook drift, clean-builds generated `dist/`, runs `tsc --noEmit`, unit/fake tests, generated command-reference blocks, and live command-reference sampling. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-18 (590 passed, 2 opt-in skips; live command-reference verification passed). |
|
|
62
63
|
| Pre-PR local gate | `npm run verify -- pre-pr` composes the default gate with package-content verification. Use before larger local handoffs or PR-ready claims when lifecycle/platform/live dogfood cost is not warranted. | **Current for 0.32.0:** covered by the stronger passing release composition on 2026-07-16; orchestration remains locked by `test/project-verify.test.ts`. |
|
|
63
|
-
| Real upstream contract | `npm run verify -- real-upstream` runs the localhost fixture matrix against the real installed `agent-browser` matching the baseline. | **Current for 0.32.2:** pass on 2026-07-
|
|
64
|
-
| Packaged Pi smoke | `npm run verify -- package-pi` validates package contents, loads the packaged `agent_browser` tool without requiring optional Brave config, and executes fake-upstream `--version`. | **Current for
|
|
65
|
-
| Startup profile | `npm run verify -- startup-profile --samples <n>` clean-builds generated `dist/`, records direct package entrypoint import/factory timing in fresh Node processes, and writes `.artifacts/startup-profile/latest.json`. It must not launch Pi, tmux, mise, npm, browsers, or `agent-browser`; full Pi TUI ready-prompt profiling is intentionally excluded after it proved too invasive for routine verification. Run this opt-in evidence when package layout, the compiled entrypoint, top-level imports, schema registration, or prompt/config startup logic changes. | **Current for compiled entrypoint:** pass on 2026-
|
|
66
|
-
| Deterministic dogfood smoke | `npm run verify -- dogfood` (`scripts/verify-agent-browser-dogfood.ts`) drives the native wrapper against a local file fixture through top-level `qa`, `semanticAction`, constrained `job`, screenshot artifact verification, and session close with the real `agent-browser` on `PATH`. | **Current for 0.32.
|
|
64
|
+
| Real upstream contract | `npm run verify -- real-upstream` runs the localhost fixture matrix against the real installed `agent-browser` matching the baseline. | **Current for 0.32.2:** pass on 2026-07-18 (2/2 real-upstream tests), including snapshot-ref native select, stable-id/label tab switching, active-target refresh after `tab close`, and visible `read` content against the localhost contract fixture. |
|
|
65
|
+
| Packaged Pi smoke | `npm run verify -- package-pi` validates package contents, loads the packaged `agent_browser` tool without requiring optional Brave config, and executes fake-upstream `--version`. | **Current for package 0.2.71 / Pi 0.80.10:** pass on 2026-07-18 (116 packed entries; exactly one packaged `agent_browser`; invocation passed). |
|
|
66
|
+
| Startup profile | `npm run verify -- startup-profile --samples <n>` clean-builds generated `dist/`, records direct package entrypoint import/factory timing in fresh Node processes, and writes `.artifacts/startup-profile/latest.json`. It must not launch Pi, tmux, mise, npm, browsers, or `agent-browser`; full Pi TUI ready-prompt profiling is intentionally excluded after it proved too invasive for routine verification. Run this opt-in evidence when package layout, the compiled entrypoint, top-level imports, schema registration, or prompt/config startup logic changes. | **Current for compiled entrypoint:** pass on 2026-07-18 (`npm run verify -- startup-profile --samples 3`; median 49.1 ms, maximum 50.1 ms, below the 250 ms budget). Full-Pi startup numbers from the unsafe tmux profiler are not accepted as ongoing release evidence. |
|
|
67
|
+
| Deterministic dogfood smoke | `npm run verify -- dogfood` (`scripts/verify-agent-browser-dogfood.ts`) drives the native wrapper against a local file fixture through top-level `qa`, `semanticAction`, constrained `job`, screenshot artifact verification, and session close with the real `agent-browser` on `PATH`. | **Current for 0.32.2:** pass on 2026-07-18; checkout-loaded Artificial Analysis/React/GitHub dogfood also passed snapshot-ref interactions, document scroll, and multi-tab close/reuse without background restarts or tab resets. |
|
|
67
68
|
| WebGPU and restore autosave | Live 0.31.2 feature probes validate the new upstream paths independently from ordinary browser dogfood. | **macOS:** `doctor --webgpu` passed Apple Metal render/readback and headless red-pixel screenshot checks; the official Hello Triangle sample exposed the Metal adapter and produced a verified non-black screenshot. **Ubuntu image:** `doctor --webgpu --headed --json` passed SwiftShader render/readback and red-pixel screenshot checks with the baked Vulkan/Mesa/Xvfb dependencies. **Restore autosave:** an idle page timer changed localStorage and the namespaced restore file contained the new value before close; test state was removed. **Windows interactive desktop:** post-release validation on 2026-07-15 used a disposable clone of snapshot `57cc3c0d-7d7c-4a4c-9c57-8270d4174679`, a logged-in console session, `agent-browser 0.31.2`, and Edge 150. The headed doctor WebGPU render/readback subcheck passed in 0.99s and its decoded screenshot subcheck passed with `rgb(255,0,0)`; a separate `--webgpu --headed` launch opened the proof page and the Parallels console capture visibly recorded the red triangle (553,500 bright-red pixels; SHA-256 `f5a28f5336cbdfeb0ff557af9425458bddc6b266d3cc6b946de7b101a0b43288`). The full doctor remained nonzero only for the unrelated absence of a separately installed Chrome binary; the explicit Edge executable completed the launch and both WebGPU probes. Local evidence is under `.artifacts/windows-webgpu-interactive/20260715T150903Z/summary.json`; the disposable clone and temporary account were deleted afterward. |
|
|
68
|
-
| Efficiency benchmark | `npm run verify -- benchmark` runs deterministic browser workflow accounting plus focused benchmark tests, including JSONL sampling fixtures and job/qa/sourceLookup/networkSourceLookup/Electron scenario coverage. | **Current:** pass on 2026-
|
|
69
|
-
| Crabbox platform smoke | `npm run check:platform-smoke` syntax-checks the harness and cheap invariants. `npm run smoke:platform:ubuntu-image` builds the project-owned Linux image, `npm run smoke:platform:doctor` checks Crabbox 0.26.0+ and local target readiness, and `npm run smoke:platform:all` runs doctor first, then fast target-local `platform-build` (`npm run verify -- platform-target`, pack, clean Pi install) plus `browser-dogfood-smoke` on Crabbox `macos`, `ubuntu`, and `windows-native`; see [`platform-smoke.md`](platform-smoke.md). Target artifacts include Crabbox/provider/work-root metadata, and release review also checks provider-specific `crabbox list` commands for leftover leases/clones. | **Current for 0.32.
|
|
70
|
-
| `verify -- release` / `prepublishOnly` | `npm run verify -- release` chains the default gate with the configured-source lifecycle harness, packaged Pi smoke, and the release-blocking Crabbox platform matrix (`verifySteps` `release` in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs)). `package.json` `prepublishOnly` runs that compose before `npm pack --dry-run` during `npm publish`. It intentionally omits standalone real-upstream, host-only dogfood, and benchmark modes—see [`RELEASE.md`](RELEASE.md#pre-release-checks). | **Current for 0.32.
|
|
71
|
-
| Configured-source lifecycle | `npm run verify -- lifecycle` (`scripts/verify-lifecycle.mjs`) drives `/reload`, closes and relaunches Pi with the same exact `--session-id`, checks the JSONL session header id, session continuity, slash-command sentinel tokens (`v1` before reload and `v2` after full relaunch because compiled JS package modules are process-cached), persisted spill reachability, and real Pi `tool_result` failure-patch semantics for a QA reclassification with a fake upstream on `PATH`. Default Pi model is `zai/glm-5.2`; default per-step wait is **180000 ms** (`DEFAULT_TIMEOUT_MS`); override model with `--model <id>` and waits with `--timeout-ms <ms>`. Passthrough flags in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs): `--keep-artifacts`, `--model`, `--verbose`, and `--timeout-ms` plus a value (for example `npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal --keep-artifacts --verbose --timeout-ms 600000`). | **Current for 0.32.
|
|
72
|
-
| Quick isolated Pi smoke | `pi --approve --no-extensions --no-skills -e . --tools agent_browser` from trusted repo root; native `agent_browser` only. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-
|
|
69
|
+
| Efficiency benchmark | `npm run verify -- benchmark` runs deterministic browser workflow accounting plus focused benchmark tests, including JSONL sampling fixtures and job/qa/sourceLookup/networkSourceLookup/Electron scenario coverage. | **Current:** pass on 2026-07-18 (`npm run verify -- benchmark`; 13/13 deterministic scenarios passed). |
|
|
70
|
+
| Crabbox platform smoke | `npm run check:platform-smoke` syntax-checks the harness and cheap invariants. `npm run smoke:platform:ubuntu-image` builds the project-owned Linux image, `npm run smoke:platform:doctor` checks Crabbox 0.26.0+ and local target readiness, and `npm run smoke:platform:all` runs doctor first, then fast target-local `platform-build` (`npm run verify -- platform-target`, pack, clean Pi install) plus `browser-dogfood-smoke` on Crabbox `macos`, `ubuntu`, and `windows-native`; see [`platform-smoke.md`](platform-smoke.md). Target artifacts include Crabbox/provider/work-root metadata, and release review also checks provider-specific `crabbox list` commands for leftover leases/clones. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-18 inside `npm run verify -- release`; evidence roots are `run-1784406200583-yaojql` (macOS), `run-1784406200585-fyulk0` (Ubuntu), and `run-1784406200586-w1nkt1` (native Windows). All platform-build, browser-dogfood, and lease-cleanup summaries passed; no release-owned leases/clones remained. |
|
|
71
|
+
| `verify -- release` / `prepublishOnly` | `npm run verify -- release` chains the default gate with the configured-source lifecycle harness, packaged Pi smoke, and the release-blocking Crabbox platform matrix (`verifySteps` `release` in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs)). `package.json` `prepublishOnly` runs that compose before `npm pack --dry-run` during `npm publish`. It intentionally omits standalone real-upstream, host-only dogfood, and benchmark modes—see [`RELEASE.md`](RELEASE.md#pre-release-checks). | **Current for 0.32.2 / package 0.2.71 / Pi 0.80.10:** pass on 2026-07-18, including 590 passing default tests, live command-reference sampling, lifecycle, packaged Pi smoke, and the required macOS/Ubuntu/native-Windows matrix. |
|
|
72
|
+
| Configured-source lifecycle | `npm run verify -- lifecycle` (`scripts/verify-lifecycle.mjs`) drives `/reload`, closes and relaunches Pi with the same exact `--session-id`, checks the JSONL session header id, session continuity, slash-command sentinel tokens (`v1` before reload and `v2` after full relaunch because compiled JS package modules are process-cached), persisted spill reachability, and real Pi `tool_result` failure-patch semantics for a QA reclassification with a fake upstream on `PATH`. Default Pi model is `zai/glm-5.2`; default per-step wait is **180000 ms** (`DEFAULT_TIMEOUT_MS`); override model with `--model <id>` and waits with `--timeout-ms <ms>`. Passthrough flags in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs): `--keep-artifacts`, `--model`, `--verbose`, and `--timeout-ms` plus a value (for example `npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal --keep-artifacts --verbose --timeout-ms 600000`). | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-18 inside `npm run verify -- release`; reload/relaunch continuity, persisted spill reachability, and failure-patch assertions passed. |
|
|
73
|
+
| Quick isolated Pi smoke | `pi --approve --no-extensions --no-skills -e . --tools agent_browser` from trusted repo root; native `agent_browser` only. | **Current for 0.32.2 / Pi 0.80.10:** pass on 2026-07-18 across isolated tmux runs using Artificial Analysis, React, GitHub, and a deterministic select/tab fixture. Snapshot-ref select/click, document scroll (0→700), stable-id/label tab switching, `tab close` target refresh, and post-close interaction passed with zero background restarts, `about:blank` resets, or spurious stale-ref failures; all disposable sessions/tmux/temp files were removed. |
|
|
73
74
|
|
|
74
75
|
Runtime floor note: package metadata keeps Pi core package peer ranges wildcard per installed Pi package docs, but `pi-agent-browser-doctor` / `npm run doctor` treats `pi --version` below 0.80.6 as a setup failure. `npm run doctor` passed against Pi 0.80.6 on 2026-07-11. This keeps package dependency shape aligned with Pi package loading while still making unsupported host Pi versions a release and first-run blocker.
|
|
75
76
|
|
|
@@ -78,11 +79,11 @@ Runtime floor note: package metadata keeps Pi core package peer ranges wildcard
|
|
|
78
79
|
| Baseline section | Baseline items | Documentation | Runtime handling | Test coverage | Validation status |
|
|
79
80
|
| --- | --- | --- | --- | --- | --- |
|
|
80
81
|
| Built-in skills | 15 canonical tokens from baseline section `skills`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#built-in-skills). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#built-in-skills), generated baseline block, README proof section, release docs. | `needsManagedSession` keeps read-only skills inspection sessionless while preserving thin upstream passthrough; upstream `@agent-browser/sandbox` remains external package guidance, not a bundled wrapper dependency. | Runtime and extension-validation skills/provider matrix; real-upstream inspection/skills group. | Supported. |
|
|
81
|
-
| Core page, element, navigation, and extraction commands | 81 canonical tokens from baseline section `core-commands`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#core-page-and-element-commands). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#core-page-and-element-commands), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md), README quick start. | Thin passthrough with wrapper-owned JSON/session planning, ref guidance, artifact verification, page-change summaries, click-dispatch diagnostics, no-op
|
|
82
|
-
| Sessions, state, tabs, frames, dialogs, and windows | 24 canonical tokens from baseline section `state-tabs-frames-dialogs`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#session-state-frames-dialogs-windows-and-inspection-commands). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#session-state-frames-dialogs-windows-and-inspection-commands), stateful workflow notes, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details). | Stateful summaries/redaction, state artifact handling, sessionless local command planning, managed-session restore, tab target pinning, and close alias cleanup. | Extension-
|
|
82
|
+
| Core page, element, navigation, and extraction commands | 81 canonical tokens from baseline section `core-commands`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#core-page-and-element-commands). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#core-page-and-element-commands), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md), README quick start. | Thin passthrough with wrapper-owned JSON/session planning, ref guidance, artifact verification, page-change summaries, click-dispatch diagnostics, deterministic document scroll before wheel/no-op diagnostics, shorthand compilers, and redaction. | Real-upstream core matrix (including snapshot-ref select) plus fake core matrix for passthrough, ordering, diagnostics, and compiler validation. | Supported. Upstream semantics remain upstream-owned. |
|
|
83
|
+
| Sessions, state, tabs, frames, dialogs, and windows | 24 canonical tokens from baseline section `state-tabs-frames-dialogs`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#session-state-frames-dialogs-windows-and-inspection-commands). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#session-state-frames-dialogs-windows-and-inspection-commands), stateful workflow notes, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details). | Stateful summaries/redaction, stable helper subprocess launch environment, state artifact handling, sessionless local command planning, managed-session restore, tab target pinning, active-target refresh after tab close, and close alias cleanup. | Extension tab/ref tests, real-upstream stable-id/label tab lifecycle, runtime session/resume tests, presentation redaction tests, lifecycle harness. | Supported. External profile/auth state remains operator-owned. |
|
|
83
84
|
| Network, storage, artifacts, diagnostics, and performance | 43 canonical tokens from baseline section `network-storage-artifacts-diagnostics`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#page-state-finding-mouse-settings-network-and-storage). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#page-state-finding-mouse-settings-network-and-storage), diagnostic sections, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details). | Thin passthrough plus compact diagnostics, route-mock warnings, useful-but-redacted storage output, stream idempotency normalization, artifact metadata, missing-ffmpeg warnings, sensitive-data redaction, timeout bounds, and cleanup-pair guidance. | Fake non-core matrix and safe real-upstream coverage for network/HAR, diff, trace/profiler, console/errors/highlight, stream, vitals, and React missing-renderer. | Supported. Environment-sensitive operations need suitable local/browser state. |
|
|
84
85
|
| Batch, auth, confirmations, setup, dashboard, devices, and AI commands | 33 canonical tokens from baseline section `batch-auth-setup-ai`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#batch-auth-confirmations-sessions-chat-dashboard-devices-and-setup). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#batch-auth-confirmations-sessions-chat-dashboard-devices-and-setup), README security notes, release docs. | Native-tool batch stdin, generated `job`/`qa`/lookup batch plans, auth/confirmation redaction, sessionless local auth/setup/dashboard/doctor/plugin planning, plugin list/show JSON envelope normalization, bare-`mcp` validation with `mcp --help` preserved, timeout/cleanup guidance. | Parser/runtime plugin and MCP unit coverage; fake-upstream plugin list/show and MCP help/blocking coverage; real-upstream plugin list shape probe; structured input-mode tests; efficiency benchmark scenarios. | Supported. Interactive side-effecting setup/auth/chat remains upstream-owned. `plugin` is local/sessionless; `mcp` is external-client-only except help; `auth login --credential-provider` resolves credentials via a plugin; `install --with-deps` failures remain upstream-owned. |
|
|
85
|
-
| Global flags, config, providers, policy, and environment | 138 canonical tokens from baseline section `options-and-env`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#important-global-flags-config-and-environment). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#important-global-flags-config-and-environment), README provider/setup notes, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#sessionmode), architecture/runtime docs. | Runtime handles command discovery, value-flag prevalidation, launch-scoped flags, redacted echoes, fresh-session recovery hints, explicit sessions, provider/device launch-scoping, parent env forwarding with
|
|
86
|
+
| Global flags, config, providers, policy, and environment | 138 canonical tokens from baseline section `options-and-env`; see [`scripts/agent-browser-capability-baseline.mjs`](../scripts/agent-browser-capability-baseline.mjs) and generated [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#important-global-flags-config-and-environment). | [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#important-global-flags-config-and-environment), README provider/setup notes, [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#sessionmode), architecture/runtime docs. | Runtime handles command discovery, value-flag prevalidation, launch-scoped flags, redacted echoes, fresh-session recovery hints, explicit sessions, provider/device launch-scoping, parent env forwarding with consistent managed idle-timeout/socket/operation-timeout overrides on top-level and helper subprocesses, subprocess completion, and package-owned Pi-scoped config for optional companion features. | Runtime tests for flags/planning/redaction/session behavior; process/ref tests for env consistency and stdio-linger completion; config/web-search/CLI tests; fake provider/specialized-skill matrix; package doctor. | Supported. Provider clouds, iOS/Appium, proxies, profiles, and credentials require external setup. |
|
|
86
87
|
|
|
87
88
|
## Follow-up decision after closure
|
|
88
89
|
|
package/docs/TOOL_CONTRACT.md
CHANGED
|
@@ -150,8 +150,8 @@ The extension always plans normal browser commands with `--json` prepended in `e
|
|
|
150
150
|
- Do not assume Playwright selector dialects such as text=Close or button:has-text('Close') are supported wrapper syntax unless current upstream agent-browser behavior has been verified.
|
|
151
151
|
- For authenticated or user-specific content explicitly requested by the user, such as feeds, inboxes, account pages, or private dashboards, use a real profile only when the user/config asks for it or profiles have been inspected; do not assume --profile Default exists on every machine. Do not use a real profile for public pages just because they are dashboards. Treat visible page content from real profiles as model-visible transcript data; use --auto-connect only if profile-based reuse is unavailable or the task is specifically about attaching to a running debug-enabled browser. If profile/user-data-dir resolution fails, stop retrying opens, run profiles and/or doctor through agent_browser, then report what the user needs to configure.
|
|
152
152
|
- Do not invent fixed explicit session names for routine tasks. Use the implicit session unless you truly need multiple isolated browser sessions in the same conversation.
|
|
153
|
-
- When using launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --enable, --executable-path, --webgpu, --init-script, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), put them on the first command for that session. If you intentionally use an explicit --session, keep using that same explicit session for follow-ups.
|
|
154
|
-
- If you already used the implicit session and now need launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --enable, --executable-path, --webgpu, --init-script, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), retry with top-level sessionMode set to fresh or pass an explicit --session for the new launch; never pass --session-mode inside args. After a successful unnamed fresh launch, later auto calls follow that new session.
|
|
153
|
+
- When using launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --enable, --executable-path, --webgpu, --init-script, --idle-timeout, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), put them on the first command for that session. If you intentionally use an explicit --session, keep using that same explicit session for follow-ups.
|
|
154
|
+
- If you already used the implicit session and now need launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --enable, --executable-path, --webgpu, --init-script, --idle-timeout, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), retry with top-level sessionMode set to fresh or pass an explicit --session for the new launch; never pass --session-mode inside args. After a successful unnamed fresh launch, later auto calls follow that new session.
|
|
155
155
|
- For WebGPU pages, use args ["--webgpu", "open", "<url>"] on a fresh local browser launch; use doctor --webgpu (or --headed on Linux/Windows capture paths) to prove rendering before trusting a non-black screenshot. WebGPU cannot be combined with --cdp, --auto-connect, or provider launches unless --webgpu false overrides an enabled config/environment default.
|
|
156
156
|
- For --allowed-domains, use a fresh local Chrome context. Upstream rejects CDP/auto-connect, profiles, restore/state replay, direct-page providers, iOS/Safari, and startup/profile Chrome args because they cannot guarantee containment; Chromium also disables RTCPeerConnection while the allowlist is active.
|
|
157
157
|
- For React introspection, launch the page with --enable react-devtools before first navigation, then use react tree, react inspect <fiberId>, sourceLookup candidates for local UI source hints, react renders start/stop, or react suspense; sourceLookup is experimental and reports confidence/evidence instead of guaranteed DOM-to-file mappings. For failed fetches and APIs, networkSourceLookup (experimental) correlates failed network requests with initiator metadata and bounded workspace URL literals—candidates only, not definitive blame. Use vitals [url] for Core Web Vitals and hydration timing, and pushstate <url> for client-side SPA navigation.
|
|
@@ -609,7 +609,7 @@ Example:
|
|
|
609
609
|
Behavior:
|
|
610
610
|
- if `args` already include `--session` (including argv compiled from optional `semanticAction.session`), upstream session choice wins
|
|
611
611
|
- `"auto"` prepends the current extension-managed active session when appropriate
|
|
612
|
-
- `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser
|
|
612
|
+
- `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser; `--idle-timeout` must equal the Pi process's configured managed idle timeout or the wrapper rejects it with restart guidance
|
|
613
613
|
- upstream `--webgpu` is a launch-scoped optional boolean: both enabled and explicit `false` values require a fresh managed launch once an implicit session exists; enabled WebGPU is local-launch-only and upstream rejects combinations with CDP, auto-connect, or providers
|
|
614
614
|
- upstream restore sessions may periodically save cookies/localStorage while open; `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` passes through unchanged (`30000` default, `0` disables periodic saves but keeps save-on-close), and those restore files remain upstream-owned rather than wrapper artifacts
|
|
615
615
|
- sessionless paths skip that injection even under `"auto"`: plain-text `--help` / `-h` / `--version` / `-V` (see the generated inspection playbook fragment below), read-only `skills list`, `skills get …`, and `skills path …`, local auth profile management (`auth save/list/show/delete/remove`), local/setup commands (`profiles`, `dashboard start/stop`, `device list`, `doctor`, `install`, `upgrade`, `session id`, `session info`, `session list`, `plugin add/list/show/run`, `mcp --help`), and targeted/all local saved-state maintenance (`state list/show`, `state clear --all`, `state clear -a`, `state clear <session-name>`, `state clean --older-than <days>`, `state rename`) keep `effectiveArgs` free of the implicit managed `--session` unless the caller supplied `--session` explicitly; successful results therefore omit `usedImplicitSession` and the extension-managed `sessionName` for those calls, while bare `mcp` server calls are rejected before spawn and root `session`, untargeted `state clear`, bare `state clean`, browser-backed `auth login`, and `state save/load` keep normal managed-session injection (`extensions/agent-browser/lib/command-policy.ts`, `needsManagedSession`; `extensions/agent-browser/lib/runtime.ts`, `buildExecutionPlan`)
|
|
@@ -807,7 +807,8 @@ Additional structured fields can appear when relevant:
|
|
|
807
807
|
- `snapshotDiff` after wrapper-side `snapshot --diff` (with or without `-i`, `--search`, `--filter`, or `--viewport`). Shape: `{ addedRefs, removedRefs, changedRefs, unchangedRefs, summary }`, comparing ref ids plus role/name metadata from the previous wrapper-tracked snapshot for the session with the newly returned full ref map. It is a quick ref-map delta, not a visual diff.
|
|
808
808
|
- `networkRequestsPageFilter` after wrapper-side `network requests --current-page`, `--current-origin`, or `--current-url`. Shape: `{ cleanArgs, currentUrl, mode, matchedRows, totalRows }`; the visible rows and `details.data.requests` / `items` / `entries` are filtered while the active session page target is read with `get url`.
|
|
809
809
|
- `richInputRecovery` after a raw `find` or compiled `semanticAction` `fill` fails with `selector-not-found` and the same current-ref diagnostic finds exact editable `searchbox` / `textbox` candidates. Shape follows `RichInputRecoveryDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, inputMethodHint, nextActionIds, summary, target }`, where each candidate has `ref`, `role`, `name`, `focusArgs`, `clickArgs`, and `reason`. Visible text appends `Rich input recovery`, and `details.nextActions` gains ids from `getAgentBrowserRichInputRecoveryNextActionIds`: `focus-current-editable-ref` / `click-current-editable-ref` (or numbered variants). These actions are bounded to focus/click/inspect-style recovery: they do not include the fill text, do not press `Enter`, and do not submit. After the right current editable ref is focused, the agent should use `keyboard inserttext` or `keyboard type` with the intended text in a separate call and submit only when explicitly required by the flow.
|
|
810
|
-
- `
|
|
810
|
+
- `scrollPage` when the wrapper moves `document.scrollingElement` directly for `scroll <up|down|left|right> [px|percent]` or `scroll to end|top`; it temporarily disables smooth scrolling so immediate before/after offsets are reliable, returns `{ request, result }`, and includes `exitCode: 0` on success. Directional document no-movement falls through to upstream wheel behavior so nested panes still work. Explicit CSS-container calls `scroll <selector> <up|down|left|right> [px|percent]` remain wrapper-handled and report `details.scrollContainer`. All scroll helper shims are skipped when startup-scoped flags are present so the requested browser/profile launches before any helper command.
|
|
811
|
+
- `scrollNoop` after a successful large **top-level** upstream scroll fallback on an existing or fresh managed session when wrapper-side read-only probes before and after the command show no change in `window.scrollX` / `window.scrollY` and no change in the sampled prominent scrollable containers. To avoid pre-launching a session without caller startup state, this probe is skipped for small pixel scrolls, calls that would create a managed session only for the probe, and invocations with startup-scoped flags such as `--profile`, `--state`, `--restore`, `--namespace`, `--session-name`, `--cdp`, providers, init scripts, or similar launch settings. Shape: `{ reason: "no-observed-scroll-position-change", message, before, after, recommendations }`; `before` / `after` include viewport dimensions, document scroll dimensions, and up to ten sampled container descriptors plus scroll offsets. Container descriptors use only sample index, tag name, and ARIA role; DOM ids/classes are intentionally not stored. This diagnostic is conservative evidence that the page-level scroll likely missed a nested pane, not proof that every app-specific region is unchanged. Visible text starts with `Scroll completed with no observed movement`, appends `Scroll diagnostic: no observed scroll movement`, sets `details.data.scrolled` to `false` / `details.data.noMovement` to `true`, and `details.nextActions` gains `inspect-after-noop-scroll` (`snapshot -i`) plus `verify-noop-scroll-visually` (`screenshot`), session-prefixed when applicable.
|
|
811
812
|
- `comboboxFocus` after a successful explicit combobox-targeted `click` / `fill` / `find … click|fill` (for example `semanticAction` with role `combobox`, including when that semantic action resolves through a current visible `@ref` before execution) when a read-only probe sees the active element is combobox-like, `aria-expanded` is explicitly present (`false` or `true`), and no visible `listbox` / `option` / menu option elements are open. Shape: `{ reason: "focused-combobox-without-visible-options", message, activeElement, visibleListboxCount, visibleOptionCount, recommendations }`; `activeElement` includes bounded role/tag/expanded/hasPopup/name metadata with normal text redaction. Visible text appends `Combobox diagnostic: focused combobox did not expose visible options`, and `details.nextActions` gains `inspect-focused-combobox` (`snapshot -i`), `try-open-combobox-with-arrow` (`press ArrowDown`), and `try-open-combobox-with-enter` (`press Enter`), session-prefixed when applicable. The diagnostic is deliberately gated to explicit combobox-targeted calls to avoid extra probes or false positives on ordinary clicks/textboxes.
|
|
812
813
|
- `recordingDependencyWarning` after a successful `record start` or `record restart` when the wrapper cannot find an executable `ffmpeg` on the Pi process `PATH`. Shape: `{ reason: "ffmpeg-missing-for-recording", dependency: "ffmpeg", command, message, recommendations }`. Visible text appends `Recording dependency warning: ffmpeg not found on PATH`. This is a non-blocking preflight warning: upstream may start recording, but `record stop` needs `ffmpeg` to encode the WebM.
|
|
813
814
|
- `selectorTextVisibility` after a **successful** upstream `get text <selector>` (standalone or inside a successful `batch`) when the wrapper’s follow-up probe finds a hazard: more than one DOM match (upstream reads the first `querySelectorAll` hit, which may be the wrong tab/panel), or the first match is hidden while at least one other match is visible (requires multiple DOM nodes so a visible peer exists; a lone hidden match is not flagged). The probe is a read-only `eval --stdin` script (`buildVisibleTextProbeScript` in `extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts`) that counts matches, applies a small visibility heuristic (`display`/`visibility`/`opacity` plus non-zero client rects), may include a redacted `firstVisibleTextPreview`, and may include up to eight `visibleCandidates` entries (`index` in `querySelectorAll`, `tagName`, optional `role`, optional redacted `textPreview`). It is **not** run for simple id selectors, page-scoped `@e…` selectors, or when the selector string is withheld because `selectorMayExposeSensitiveLiteral` would risk echoing secrets in probe output. `details.selectorTextVisibility` mirrors the primary diagnostic (first sorted entry); when several selectors in one `batch` qualify, `selectorTextVisibilityAll` lists every diagnostic sorted so hidden-first cases precede generic multi-match ambiguity. Appended visible warning text names the matching `details.nextActions` id and may list visible candidate previews. Appended `details.nextActions` use ids `inspect-visible-text-candidates` and `inspect-visible-text-candidates-2`, … with the probe replayed via `eval --stdin` for each hazardous selector. If the probe still leaves more than one visible candidate, it is only ambiguity evidence; agents should narrow the selector, use a current visible `@ref`, or run a targeted visible-element `eval --stdin` rather than trusting the broad selector.
|
|
@@ -871,7 +872,7 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
871
872
|
- treat the extension-managed session as convenience state owned by the wrapper
|
|
872
873
|
- preserve the current branch-visible extension-managed session across `/reload`, exact-session relaunch, `/resume`, and Pi `session_tree` branch transitions so persisted sessions can keep following the live browser after lifecycle changes
|
|
873
874
|
- close the active extension-managed session when the originating `pi` process quits, while leaving explicit caller-provided sessions alone
|
|
874
|
-
- set
|
|
875
|
+
- set one idle timeout on extension-managed sessions as a backstop for abnormal exits or cleanup failures, and pass that same `AGENT_BROWSER_IDLE_TIMEOUT_MS` to top-level commands plus every wrapper helper subprocess so upstream does not restart the background browser, reset the active tab, or discard current refs between a snapshot and action
|
|
875
876
|
- clean up process-private temp spill artifacts on shutdown, while keeping persisted-session snapshot spill files in a private session-scoped artifact directory so `details.fullOutputPath` survives reload/restart and the oldest spill files are evicted if the per-session artifact budget is exceeded
|
|
876
877
|
- reconstruct the current branch-visible extension-managed session, latest page-scoped refs, latest `artifactManifest`, and wrapper-tracked Electron launch records from the active transcript branch on `session_start` and Pi `session_tree` so later default calls keep following the active managed browser and can continue reporting artifact retention state; successful explicit wrapper-owned close rows and `electron.cleanup` managed-session steps are restore-visible close events
|
|
877
878
|
- keep runtime cleanup ownership separate from branch-visible state: `session_tree` restore and wrapper-owned browser commands are serialized with managed-session work; independent caller-owned explicit-session commands can still run in parallel, but a branch-state generation guard prevents stale completions from overwriting a newer branch restore. Extension-managed sessions and wrapper-launched Electron records owned by the current process remain eligible for quit/cleanup, and fresh-session allocation stays monotonic across branch restores, including auto rows and close rows that reference wrapper-generated fresh names
|
|
@@ -893,7 +894,7 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
893
894
|
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms>` or `wait --timeout <ms>` calls can exceed that default; when top-level `timeoutMs` is omitted, the wrapper derives a per-call subprocess watchdog from the requested wait duration plus a small grace window. Dialog commands use `PI_AGENT_BROWSER_DIALOG_PROCESS_TIMEOUT_MS` (default 5000 ms), and click/tap/find refs or tokens plus `eval --stdin` snippets whose text looks like alert/confirm/prompt/dialog triggers use `PI_AGENT_BROWSER_DIALOG_TRIGGER_PROCESS_TIMEOUT_MS` (default 8000 ms). Timed-out compiled `job` / `qa` or caller `batch` calls may add `details.timeoutPartialProgress` and visible `Timeout partial progress` evidence with per-step status, retry payloads, current page title/URL, and declared artifact path checks; timed-out dialog-like commands may add dialog status/dismiss/fresh-session recovery next actions
|
|
894
895
|
- interactive or long-running upstream families such as `chat` without a prompt, `dashboard start`, `stream enable`, `trace start`, `profiler start`, `record start`, `inspect`, `install`, `upgrade`, `doctor --fix`, and `confirm-interactive` are passed through thinly but remain bounded by the same wrapper timeout/session planning rules; prefer explicit arguments, single-shot `chat <message>`, non-interactive flags like `doctor --offline --quick` or `doctor --json`, and cleanup pairs such as `dashboard stop`, `stream disable`, `trace stop`, `profiler stop`, and `record stop`
|
|
895
896
|
- treat successful plain-text inspection commands like `--help` and `--version` as stateless: do not inject the implicit managed session and do not let those calls claim the managed-session slot
|
|
896
|
-
- if startup-scoped flags like `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--enable`, `-p` / `--provider`, or iOS `--device` are supplied after the implicit session is already active while `sessionMode` is `"auto"`, return a validation error with a structured recovery hint that recommends `sessionMode: "fresh"`
|
|
897
|
+
- if startup-scoped flags like `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--enable`, `-p` / `--provider`, or iOS `--device` are supplied after the implicit session is already active while `sessionMode` is `"auto"`, return a validation error with a structured recovery hint that recommends `sessionMode: "fresh"`
|
|
897
898
|
- for direct headless local Chrome launches to `chat.com` / `chatgpt.com` / `chat.openai.com`, allow a narrow compatibility fallback that injects a normal Chrome `--user-agent` only when the caller did not explicitly provide one and did not choose `--headed`, `--cdp`, `--auto-connect`, or a provider-backed launch
|
|
898
899
|
|
|
899
900
|
## Non-goals
|
package/package.json
CHANGED