pi-agent-browser-native 0.9.4 → 0.9.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/README.md +58 -813
- package/dist/extensions/agent-browser/index.js +927 -883
- package/dist/extensions/agent-browser/lib/browser-execution-claims.js +5 -5
- package/dist/extensions/agent-browser/lib/config-policy.js +1 -1
- package/dist/extensions/agent-browser/lib/config-validation.js +21 -8
- package/dist/extensions/agent-browser/lib/config.js +1 -1
- package/dist/extensions/agent-browser/lib/electron/cdp.js +5 -1
- package/dist/extensions/agent-browser/lib/electron/cleanup.js +13 -47
- package/dist/extensions/agent-browser/lib/electron/discovery-evidence.js +1 -0
- package/dist/extensions/agent-browser/lib/electron/launch.js +12 -1
- package/dist/extensions/agent-browser/lib/fs-utils.js +1 -9
- package/dist/extensions/agent-browser/lib/input-modes/job.js +7 -4
- package/dist/extensions/agent-browser/lib/input-modes/lookup-network-evidence.js +15 -3
- package/dist/extensions/agent-browser/lib/input-modes/lookups.js +3 -2
- package/dist/extensions/agent-browser/lib/input-modes/params.js +23 -23
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/electron-observation.js +9 -8
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result-electron.js +8 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/scroll-shims.js +2 -2
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/snapshot-filter.js +9 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-electron.js +4 -2
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state-refs.js +9 -11
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/branch.js +1 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe-collection.js +5 -6
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe-data.js +4 -2
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/probe-presentation.js +20 -2
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/status.js +4 -2
- package/dist/extensions/agent-browser/lib/orchestration/execution-queue.js +1 -1
- package/dist/extensions/agent-browser/lib/results/presentation/skills.js +7 -2
- package/dist/extensions/agent-browser/lib/temp-artifact-budget.js +4 -1
- package/dist/extensions/agent-browser/lib/temp-root.js +22 -18
- package/dist/extensions/agent-browser/lib/temp.js +1 -1
- package/dist/extensions/agent-browser/lib/web-search-providers.js +0 -36
- package/dist/extensions/agent-browser/lib/web-search-request.js +20 -26
- package/dist/extensions/agent-browser/lib/web-search-schema.js +5 -5
- package/dist/extensions/agent-browser/lib/web-search.js +22 -4
- package/docs/ARCHITECTURE.md +3 -2
- package/docs/CODE_QUALITY.md +28 -3
- package/docs/COMMAND_REFERENCE.md +20 -20
- package/docs/ELECTRON.md +24 -2
- package/docs/REFERENCE.md +869 -0
- package/docs/RELEASE.md +9 -9
- package/docs/REQUIREMENTS.md +1 -1
- package/docs/SUPPORT_MATRIX.md +9 -8
- package/docs/TOOL_CONTRACT.md +13 -9
- package/docs/platform-smoke.md +3 -3
- package/package.json +3 -2
- package/scripts/config.mjs +23 -19
- package/scripts/doctor.mjs +2 -2
- package/scripts/platform-smoke/lease-evidence.mjs +2 -4
- package/scripts/platform-smoke/linux-image/Dockerfile +8 -2
- package/scripts/platform-smoke/targets.mjs +1 -5
package/README.md
CHANGED
|
@@ -1,871 +1,116 @@
|
|
|
1
1
|
# pi-agent-browser-native
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
pi-agent-browser-native adds browser tools to [Pi](https://pi.dev). Use it to read pages, test web apps, and save screenshots through [agent-browser](https://agent-browser.dev/).
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## Install and start
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
Use Node 24.21.0 or later and Pi 1.0.0 or later. This extension requires a separate agent-browser installation.
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
For the 0.6.16 compatibility rollout, the owner explicitly waived Windows qualification on 2026-09-21. Windows diagnostics and failures remain visible but nonblocking; this is **not** a Windows full-suite pass. Linux/macOS, declared Node floors, official/fork host and consumer-artifact checks remain required. Known Windows failures and follow-up are tracked in [#191](https://github.com/fitchmultz/pi-agent-browser-native/issues/191); see the [current support evidence](docs/SUPPORT_MATRIX.md#current-0616-rollout).
|
|
14
|
-
|
|
15
|
-
## Source-of-truth map
|
|
16
|
-
|
|
17
|
-
Start here for install and common usage. For deeper work, use the active docs by purpose:
|
|
18
|
-
|
|
19
|
-
| Need | Read |
|
|
20
|
-
| ---------------------------------------------------------------- | -------------------------------------------------------- |
|
|
21
|
-
| Command workflows and upstream CLI coverage | [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md) |
|
|
22
|
-
| Native tool input/output contract and `details` fields | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md) |
|
|
23
|
-
| Runtime design and package config policy | [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) |
|
|
24
|
-
| Browser replay, large-history limits and stopped-copy conversion | [`docs/SESSION_STORAGE.md`](docs/SESSION_STORAGE.md) |
|
|
25
|
-
| Electron desktop lifecycle | [`docs/ELECTRON.md`](docs/ELECTRON.md) |
|
|
26
|
-
| Release gates and targeted upstream support | [`docs/SUPPORT_MATRIX.md`](docs/SUPPORT_MATRIX.md) |
|
|
27
|
-
| Maintainer release process | [`docs/RELEASE.md`](docs/RELEASE.md) |
|
|
28
|
-
|
|
29
|
-
The complete documentation ownership map lives in the repository source at [`docs/SOURCE_OF_TRUTH.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/docs/SOURCE_OF_TRUTH.md).
|
|
30
|
-
|
|
31
|
-
Embedding Pi in another host? The extension factory accepts an optional awaited [`beforeExecute` callback](docs/TOOL_CONTRACT.md#host-execution-hook) for saving host state before browser dispatch. Ordinary Pi installation needs no callback.
|
|
32
|
-
|
|
33
|
-
## What this looks like in Pi
|
|
34
|
-
|
|
35
|
-
You prompt the agent in plain English:
|
|
36
|
-
|
|
37
|
-
```text
|
|
38
|
-
Use the agent_browser tool to open https://react.dev and then take an interactive snapshot.
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
The agent gets a native tool, not a bash workaround:
|
|
42
|
-
|
|
43
|
-
```json
|
|
44
|
-
{ "args": ["open", "https://react.dev"] }
|
|
45
|
-
{ "args": ["snapshot", "-i"] }
|
|
46
|
-
{ "args": ["find", "text", "Learn React", "click"] }
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
Use native commands for individual actions, `batch --bail` for fixed sequences, and `agent_browser_code` for loops or branches. Specialized tools are available through `agent_browser_tools`; see [common calls](#common-agent-calls). On hosts supporting instruction groups, enable `browser` through `discover_tools` first to receive the complete browser guidance and reveal selected declarations. This does not grant callable permissions or activate advanced tools. The extension owns group membership and one full instruction source; official Pi receives that same guidance eagerly and keeps the ordinary direct/code/loader surface.
|
|
50
|
-
|
|
51
|
-
The result is optimized for agent work:
|
|
52
|
-
|
|
53
|
-
- compact page snapshots that lead with useful page content instead of chrome/sidebar noise
|
|
54
|
-
- interactive `@eN` refs for follow-up clicks and form fills
|
|
55
|
-
- screenshots and downloaded files surfaced as Pi artifacts
|
|
56
|
-
- structured details for titles, URLs, saved files, sessions, and errors
|
|
57
|
-
- spill files for full redacted output instead of dumping oversized pages into context
|
|
58
|
-
- compact, colorized Pi TUI rows that can be expanded without changing what the agent receives
|
|
59
|
-
- recovery hints when a tab, selector, stale `@ref`, or launch mode needs a different next step
|
|
60
|
-
|
|
61
|
-
## Who this is for
|
|
62
|
-
|
|
63
|
-
- **Pi users** who want browser automation available as a normal tool beside `read`, `write`, and `bash`.
|
|
64
|
-
- **Coding agents** that need low-context browser workflows for docs, QA, research, dashboards, provider-backed browsers, and web apps.
|
|
65
|
-
- **Maintainers** who want a thin integration that tracks the current upstream [`agent-browser`](https://agent-browser.dev/) CLI without bundling or re-implementing it.
|
|
66
|
-
|
|
67
|
-
## The problem
|
|
68
|
-
|
|
69
|
-
`agent-browser` is powerful, but plain CLI use is awkward inside an agent harness:
|
|
70
|
-
|
|
71
|
-
- shell strings are easy for agents to quote wrong
|
|
72
|
-
- large page snapshots can waste model context
|
|
73
|
-
- screenshots and downloads need artifact metadata, not just text paths
|
|
74
|
-
- implicit browser sessions need predictable reuse and cleanup
|
|
75
|
-
- profile/debug launches need a clear way to start fresh after public browsing
|
|
76
|
-
- secrets and auth material must not be echoed into model-visible output
|
|
77
|
-
- stale element refs need actionable recovery guidance, not generic failures
|
|
78
|
-
|
|
79
|
-
`pi-agent-browser-native` keeps upstream `agent-browser` as the browser engine and adds the Pi-native wrapper behavior needed for reliable agent use.
|
|
80
|
-
|
|
81
|
-
## What it does
|
|
82
|
-
|
|
83
|
-
| Pain | Native wrapper capability | Proof surface |
|
|
84
|
-
| --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
85
|
-
| Agents build fragile shell commands or repeat browser calls for loops and branches | Exposes compact `agent_browser` for native commands and batch, `agent_browser_code` for fresh JavaScript against the same persistent browser, and discoverable action, QA, Electron, and source tools | `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/input-modes/`, [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md) |
|
|
86
|
-
| Page snapshots are too large or viewport-blind | Shows compact, main-content-first summaries, surfaces an `Omitted high-value controls` section (plus `details.data.highValueControlRefIds`) when dense pages or desktop host screens hide editables, named surfaces/tabs, primary action buttons, and high-signal named links such as repository results from the trimmed ref lists, supports wrapper-side `snapshot -i --search <text>` / `--filter role=<role>` to trim dense pages while preserving full `details.refSnapshot`, supports `snapshot --viewport` for scroll/viewport metadata, supports `snapshot --diff` for quick ref-map deltas versus the prior tracked snapshot, and stores full redacted output in spill files when needed | `extensions/agent-browser/lib/results/snapshot.ts`, `extensions/agent-browser/lib/orchestration/browser-run/prepare.ts`, `test/agent-browser.presentation.test.ts`, `test/agent-browser.extension-validation.test.ts` |
|
|
87
|
-
| Screenshots/downloads get lost in text | Normalizes artifact paths, creates missing parent directories, leaves click/download execution to native download, and reports existence, size, cwd, session, and repair status | [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#download-screenshot-and-pdf-files) |
|
|
88
|
-
| Profile restores and tab drift confuse agents | Tracks managed sessions, keeps every upstream helper probe on the same idle-timeout launch configuration so the background browser is not restarted between a snapshot and action, retains native CDP target IDs so same-address tabs remain distinct after resume, re-selects target tabs after observed drift, live-verifies the active target and refreshes its title after successful tab selection or `tab close` (while retaining new blank tabs, deliberate `about:blank` selections, and post-close blank targets, including batch transitions), rehydrates branch-backed session state on Pi session-tree changes, and pins later commands only for sessions with drift/restored-session risk | generated tab-recovery notes below; `test/agent-browser.extension-tab-recovery.test.ts` (drift and about:blank recovery), `test/agent-browser.extension-tabs.test.ts` (post-close target), `test/agent-browser.extension-ref-guards.test.ts` (snapshot/action environment and session-tree rehydration), `test/agent-browser.resume-state.test.ts` (persisted session / resume planning) |
|
|
89
|
-
| Auth/profile workflows can leak secrets | Supports `auth save --password-stdin`, redacts sensitive args, SAML/OAuth-bearing URLs, stdout/stderr, details, and snapshot spills, and discards malformed oversized stdout instead of persisting a parse-failure spill | `test/agent-browser.extension-security-redaction.test.ts` |
|
|
90
|
-
| Stateful cookies/storage/auth output bloats or leaks context | Presentation layer redacts `details.data` for cookies and credential-like storage values while keeping low-risk local QA values such as `theme: dark` readable; recursively scrubs other structured upstream JSON (network, diff, trace/profiler, stream, dashboard, chat, auth, dialog, frame, state, and similar) using sensitive key names plus string heuristics; masks sensitive argv flags and positionals, including raw batch rows and ordered code journals; scrubs secrets from failed batch step errors; and exposes a compact redacted `batch` matrix on top-level `details.data` | `extensions/agent-browser/lib/results/presentation.ts`, `extensions/agent-browser/lib/results/presentation/diagnostics.ts`, `extensions/agent-browser/lib/runtime.ts`, `test/agent-browser.presentation-diagnostics.test.ts` |
|
|
91
|
-
| Stale `@eN` refs fail mysteriously | Defines full per-session refs once in the native journal, exposes them on dedicated captures, rejects mismatched URLs / unknown refs / unsafe `batch` stdin ordering before spawn, adds recovery guidance to rerun `snapshot -i` or use stable `find` locators | `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/session-page-state.ts`, `test/agent-browser.session-page-state.test.ts`, `test/agent-browser.results.test.ts`, `test/agent-browser.extension-ref-guards.test.ts`, `test/agent-browser.extension-semantic-recovery.test.ts` |
|
|
92
|
-
| Agents need stable success/failure buckets | Exposes bounded `resultCategory`, `successCategory`, and `failureCategory` on tool `details` for branching without parsing prose; final results return native `isError` and a visible prose failure notice while preserving parseable caller-requested `--json` output. Native structured results expose the same bounded, redacted observations, not internal replay metadata | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/results/categories.ts`, `extensions/agent-browser/index.ts`, `extensions/agent-browser/lib/pi-tool-rendering.ts`, `test/agent-browser.results.test.ts`, `test/agent-browser.extension-validation.test.ts`, `test/agent-browser.pi-pipeline.test.ts` |
|
|
93
|
-
| Clicks can report success without the page receiving the event | Top-level non-Electron direct `click` calls on `xpath=` targets or role-gated current `@e…` refs (`button`, `checkbox`, `menuitem`, `radio`, `switch`, `tab`) install a bounded target-specific DOM-event probe; eligible `@e…` refs require a unique role/name in both the latest snapshot and the live candidates. A per-probe temporary DOM marker must be read back by native `get attr <original-selector> <marker>` before the candidate can verify dispatch, including XPath scope. Missing or mismatched identity cleans up the probe and leaves the native click unverified, not failed. Normal collection cancels the probe timer; if a pending native decision blocks cleanup, listeners and the window marker self-expire after this call's process-timeout budget plus the existing two-second cleanup grace, without another browser command. Duplicate-name refs pass through without a probe: their old ordinal cannot prove which element received the click. If upstream reports success but no trusted event reaches the resolved target, the wrapper fails the tool, exposes `details.clickDispatch`, and suggests explicit retry/inspect next actions (no in-page replay), including a nested-scroll `scrollintoview` action when the probe sees the target outside a scroll container or viewport. Unresolved locator clicks such as raw `find … click` are left upstream-owned to avoid false failures for frame-scoped targets. Other click results still expose `details.pageChangeSummary`; `observed: false` explicitly marks dispatch-only mutation summaries and adds a visible `Action dispatched; application change unverified` warning. Unchanged-URL clicks can surface evidence-backed `details.overlayBlockers` candidates. | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), `extensions/agent-browser/lib/orchestration/browser-run/click-dispatch.ts`, `extensions/agent-browser/lib/results/presentation/navigation.ts`, `test/agent-browser.presentation.test.ts`, `test/agent-browser.extension-click-dispatch.test.ts` |
|
|
94
|
-
| Dashboard scroll commands can look successful while nothing moves | Handles standard `scroll <dir> [px]` against the document first (including pages whose smooth-scroll CSS defeats upstream wheel timing), falls back upstream when the document cannot move, and samples viewport/containers around the fallback; explicit CSS-container scrolling uses instant movement so smooth-scroll CSS cannot cause a premature no-movement failure; unchanged positions fail as `upstream-error` with `details.scrollNoop`, visible recovery guidance, and exact snapshot/screenshot checks. Unsupported `scrollintoview text=...` fails before dispatch, including inside effective batch rows, and shows exact native `find text ... hover` and snapshot/ref recovery payloads; help remains native pass-through. | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#core-page-and-element-commands), `test/agent-browser.extension-validation.test.ts` |
|
|
95
|
-
| Dropdown/combobox clicks can focus or hit native option box-model errors | Adds first-class `select <selector> <value...>` paths through native `args` and `agent_browser_action`; semantic role/name or label select resolves exactly one current visible combobox/listbox ref before action. Custom combobox clicks still detect focused controls with explicit `aria-expanded` state but no visible options and return `details.comboboxFocus` plus exact recovery `nextActions` | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#core-page-and-element-commands), `extensions/agent-browser/lib/input-modes/semantic-action.ts`, `test/agent-browser.extension-input-modes.test.ts`, `test/agent-browser.extension-validation.test.ts` |
|
|
96
|
-
| Recording workflows fail late when `ffmpeg` is missing or report stale lifecycle state | After successful `record start` / `record restart`, reports `successCategory: "artifact-pending"`, returns an exact `stop-pending-recording` action, warns when older natives report pending output without `ffmpeg`, and conservatively invalidates prior page-scoped `@e…` refs on every executed start attempt and URL-bearing restart to protect older supported natives. This is not evidence of a page change: 0.37 records the active page unless a URL is supplied; FPS-only calls keep the intended tab; an unbounded transcript-backed namespace/session index reserves active destinations across aliases, serializes artifact lifecycle and explicit wait/output writes, persists cross-branch close tombstones, retires every successful close path (including every matching namespace owner for `close --all`), rejects missing/stale restart output, coalesces terminal batch state, keeps only the newest pending path per identity, rejects recording starts after a nested close, folds Unicode path aliases, and retains exact cleanup actions with visible guidance on any later same-session failure | [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#details), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#diff-debug-and-streaming), `test/agent-browser.extension-validation.test.ts`, `test/agent-browser.presentation-artifacts-batch.test.ts` |
|
|
97
|
-
| Upstream CLI drift can silently invalidate wrapper behavior | Publishes a repo-readable command reference, verifies it against the recommended 0.38.1 target, and probes browser-backed calls once per cwd/PATH so stable versions below the 0.35.0 floor fail before browser launch with installed/expected version evidence | `npm run verify` |
|
|
98
|
-
| Pages can expose structured workflows through experimental WebMCP | Shows native positive WebMCP availability in navigation summaries and passes through `webmcp list`, `invoke`, detached `result` / `cancel`, params/frame/timeout options, and the bundled `webmcp-gen` skill; treats `--no-webmcp` as launch-scoped, keeps pending or unsuccessfully settled targets unverified with an actionable `get url` follow-up, invalidates stale refs after page tools run, and budgets effective raw or stdin batch timeouts | [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#webmcp-page-tools), `test/agent-browser.extension-ref-guards.test.ts`, `test/agent-browser.wait-timeouts.test.ts`, `test/agent-browser.real-upstream-contract.test.ts` |
|
|
99
|
-
| Desktop Electron apps need discovery, CDP attach, and safe teardown | `agent_browser_electron` runs host `list` / isolated `launch` (temp profile, OS-chosen debug port) / `status` / `probe` / `cleanup`, merges `launchId` plus managed `sessionName`, supports `handoff` `snapshot` / `tabs` / `connect`, and surfaces mismatch and post-command health guidance; wrapper cleanup applies only to launches it created | `extensions/agent-browser/lib/electron/discovery.ts`, `launch.ts`, `cleanup.ts`, [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md#agent_browser_electron), [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#electron-desktop-apps) |
|
|
100
|
-
| Agents need bundled `skills` text and local setup/status commands without touching the live session | Treats `skills list/get/path`, local auth/profile/setup commands, `session list`, and local state lifecycle commands as sessionless reads/actions when upstream does not need a live page. Session/state rows and targets remain visible, and supported upstream state/config/path operations pass through unchanged. Browser-backed workflows still receive an implicit session only when the caller did not choose one. | [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md#built-in-skills), `extensions/agent-browser/lib/command-policy.ts`, `extensions/agent-browser/lib/runtime.ts` |
|
|
101
|
-
|
|
102
|
-
Artifact results show known requested paths separately from reported/resolved locations and infer image MIME types from file headers, not suffixes. Parent-directory failures return path-specific `validation-error` guidance before browser dispatch. Prefer absolute artifact paths in raw batch strings because the daemon's working directory may differ from Pi's. When execution cwd differs from the launch/project root, the wrapper binds relative file operands, including raw batch rows, to the captured execution cwd. Dispatched `record start` and URL-bearing `record restart` attempts also return one fresh-snapshot warning on success or failure, including in JSON output. It describes conservative ref invalidation, not an observed page change; unreached rows do not emit it.
|
|
103
|
-
|
|
104
|
-
## Fastest way to try it
|
|
105
|
-
|
|
106
|
-
Use Node 24.21.0 or newer and Pi 1.0.0 or newer. This package keeps optional Pi core imports as wildcard `peerDependencies` because Pi package docs require the host Pi install to provide those packages, retains a locked Pi development snapshot while qualifying the latest official/fork graphs, and makes Pi hosts below 1.0.0 a setup failure through `pi-agent-browser-doctor`. There are no compatibility shims for older Pi releases.
|
|
107
|
-
|
|
108
|
-
Install upstream `agent-browser` first and make sure it is on `PATH`:
|
|
109
|
-
|
|
110
|
-
- https://agent-browser.dev/
|
|
111
|
-
- https://github.com/vercel-labs/agent-browser
|
|
112
|
-
|
|
113
|
-
Optional external tools unlock the full command surface:
|
|
114
|
-
|
|
115
|
-
| Dependency | Required for | macOS install example |
|
|
116
|
-
| --------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------- |
|
|
117
|
-
| `agent-browser` | All browser automation through this extension | See upstream install docs |
|
|
118
|
-
| `ffmpeg` | Recording WebM (libvpx) or MP4 (libx264); install before `record start` / `record restart` | `brew install ffmpeg` or `brew install ffmpeg-full` |
|
|
119
|
-
|
|
120
|
-
Keep both binaries on `PATH`. This package recommends `agent-browser 0.38.1` and accepts stable versions at or above the 0.35.0 floor; browser-backed calls fail fast below that floor while local inspection/setup commands remain available for diagnosis. Native 0.37 validates `ffmpeg`, the output extension and FPS before recording; older supported natives may defer failures until stop. Use `record start <path.webm|path.mp4> [url] [--fps <n>]` (1–60 fps, default 30), then verify the completed file after `record stop`. Native `doctor` checks the recording dependency and encoders.
|
|
121
|
-
|
|
122
|
-
On 0.38.1, use `snapshot --delta` for native full/unchanged/changed revisions and `snapshot --delta --full` to reset its baseline. Surviving DOM nodes keep their refs, but navigation/replaced nodes still require fresh refs. The wrapper preserves native revisions and refreshes complete refs internally for partial results. `screenshot --if-changed` (or `--threshold 0.01`) skips unchanged images without creating an artifact. `--input-mode instant|smooth|human` controls native session pointer movement; `click ... --human`, `drag ... --human`, and timed `mouse move` remain ordinary `args`. Recordings accept `--cursor` and `--contact-sheet`; the latter produces a timestamped PNG verified after stop. `auth login <name> --no-navigate` fills a prepared login page after upstream checks its origin. See the [command reference](docs/COMMAND_REFERENCE.md#upstream-0381-rebaseline) for examples and limits.
|
|
123
|
-
|
|
124
|
-
### Android / Termux
|
|
125
|
-
|
|
126
|
-
Git/source installs and type checks require an **Android-native TypeScript 7 compiler**
|
|
127
|
-
available as `tsgo` on `PATH`. The [Fold development environment](https://github.com/fitchmultz/fold-dev-environment)
|
|
128
|
-
provides a pinned native build (validated here with `7.1.0-dev.20260904.1`).
|
|
129
|
-
The npm TypeScript package has no Android binary, and its Linux binary probes a
|
|
130
|
-
syscall blocked by Android. Build/type-check scripts therefore use native `tsgo`
|
|
131
|
-
on Termux; macOS/Linux continue using the package-local pinned `tsc`.
|
|
132
|
-
Published precompiled npm packages do not require a compiler at runtime.
|
|
133
|
-
|
|
134
|
-
Android support currently uses Termux's system Chromium rather than Chrome for Testing. Upstream issue [vercel-labs/agent-browser#1587](https://github.com/vercel-labs/agent-browser/issues/1587) tracks native Android packaging; until upstream ships an Android launcher, install the packaged Linux-musl arm64 binary without lifecycle scripts and point the global command at it. The last locally validated Android setup used 0.36.0:
|
|
135
|
-
|
|
136
|
-
```bash
|
|
137
|
-
pkg install tur-repo x11-repo
|
|
138
|
-
pkg install chromium ffmpeg which
|
|
139
|
-
npm install -g --ignore-scripts agent-browser@0.36.0
|
|
140
|
-
ln -sfn "$(npm root -g)/agent-browser/bin/agent-browser-linux-musl-arm64" \
|
|
141
|
-
"$(npm prefix -g)/bin/agent-browser"
|
|
142
|
-
ln -sfn "$PREFIX/lib/chromium/chromium-launcher.sh" "$PREFIX/bin/chromium"
|
|
143
|
-
agent-browser --version
|
|
144
|
-
which chromium
|
|
145
|
-
```
|
|
146
|
-
|
|
147
|
-
The `which` package and launcher symlink satisfy upstream's existing Linux system-browser discovery even when tests or Pi sessions isolate `HOME`. Browser code uses the same native launch configuration as direct calls.
|
|
148
|
-
|
|
149
|
-
Reapply the musl command symlink after reinstalling or upgrading upstream until #1587 is resolved. The wrapper uses Termux-private socket/policy storage, compact 80-bit managed identities so ordinary namespaces and fresh rotations fit the Unix socket-path limit, Termux's `ps`, and Android app-sandbox trust rules automatically. Historical 0.6 Android validation covered headless browser flows, managed restore, namespaced sessions, the former script/job inputs, QA, screenshots, and recording; it does not qualify the 0.7 code contract. Electron desktop discovery/lifecycle is not applicable to Android apps. Android remains outside the release-blocking Crabbox macOS/Ubuntu/native-Windows matrix until a repeatable Android provider target is added.
|
|
150
|
-
|
|
151
|
-
The native tool also gives agents absolute installed-package doc paths in its full browser instructions. Raw `args` are the 1:1 upstream CLI coverage path for the targeted `agent-browser` release; `agent_browser_code` adds bounded JavaScript orchestration over the same browser, while advanced tools preserve semantic actions, diagnostic QA, source lookup, and Electron lifecycle. Agents should read `README.md` for setup/dependencies, `docs/COMMAND_REFERENCE.md` for targeted command workflows, and `docs/TOOL_CONTRACT.md` for result/detail contracts only when deeper guidance is needed.
|
|
152
|
-
|
|
153
|
-
Then install this Pi package:
|
|
11
|
+
The recommended agent-browser version is 0.38.1. This extension accepts stable versions 0.35.0 or later.
|
|
154
12
|
|
|
155
13
|
```bash
|
|
14
|
+
npm install -g agent-browser
|
|
15
|
+
agent-browser install
|
|
156
16
|
pi install npm:pi-agent-browser-native
|
|
17
|
+
pi
|
|
157
18
|
```
|
|
158
19
|
|
|
159
|
-
|
|
20
|
+
`agent-browser install` downloads Chrome for Testing. On Linux, use `agent-browser install --with-deps` if browser libraries are missing. Android users need the [Termux setup](docs/REFERENCE.md#android--termux).
|
|
160
21
|
|
|
161
|
-
|
|
22
|
+
Ask Pi:
|
|
162
23
|
|
|
163
24
|
```text
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
```bash
|
|
170
|
-
pi --no-extensions -e npm:pi-agent-browser-native
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
`--no-extensions` disables automatic extension loading, not Pi settings, configured package resolution, skills, prompts, themes, or context files.
|
|
174
|
-
|
|
175
|
-
Pi may ask whether to trust projects with trust-gated settings or resources. This extension follows Pi's trust decision when loading its project-local config. `--no-approve` skips that config and Pi's trust-gated project resources; context files such as `AGENTS.md` still load unless context loading is separately disabled.
|
|
176
|
-
|
|
177
|
-
For a specific published version:
|
|
178
|
-
|
|
179
|
-
```bash
|
|
180
|
-
pi --no-extensions -e npm:pi-agent-browser-native@<version>
|
|
181
|
-
```
|
|
182
|
-
|
|
183
|
-
To install directly from source instead of npm:
|
|
184
|
-
|
|
185
|
-
```bash
|
|
186
|
-
pi install https://github.com/fitchmultz/pi-agent-browser-native
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
For a source trial without adding the package to your Pi settings:
|
|
190
|
-
|
|
191
|
-
```bash
|
|
192
|
-
pi --no-extensions -e https://github.com/fitchmultz/pi-agent-browser-native
|
|
25
|
+
Open https://example.com with agent_browser.
|
|
26
|
+
Take an interactive snapshot.
|
|
27
|
+
Save a screenshot to /tmp/example.png.
|
|
28
|
+
Close the browser when finished.
|
|
193
29
|
```
|
|
194
30
|
|
|
195
|
-
|
|
31
|
+
After a package update, quit Pi fully. Start Pi again to load the new tools. `/reload` can retain the previous compiled code.
|
|
196
32
|
|
|
197
|
-
|
|
33
|
+
Read the [command reference](docs/COMMAND_REFERENCE.md) for your next browser task.
|
|
198
34
|
|
|
199
|
-
|
|
200
|
-
pi-agent-browser-doctor
|
|
201
|
-
# one-off without permanent install:
|
|
202
|
-
npm exec --package pi-agent-browser-native -- pi-agent-browser-doctor
|
|
203
|
-
# from this checkout:
|
|
204
|
-
npm run doctor
|
|
205
|
-
```
|
|
35
|
+
## Browser tools
|
|
206
36
|
|
|
207
|
-
|
|
37
|
+
Describe the task in your conversation. Pi can click controls, fill forms, check text and URLs, and save files.
|
|
208
38
|
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
39
|
+
| Tool | Use |
|
|
40
|
+
| -------------------------- | ------------------------------------------------------------- |
|
|
41
|
+
| `agent_browser` | Browser commands and fixed `batch --bail` sequences |
|
|
42
|
+
| `agent_browser_code` | JavaScript loops and branches in the same browser |
|
|
43
|
+
| `agent_browser_tools` | Enable action, QA, Electron, source, and network-source tools |
|
|
44
|
+
| `agent_browser_web_search` | Optional Exa or Brave search through provider APIs |
|
|
213
45
|
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
Pi hosts that run as uid 0 should set `PI_AGENT_BROWSER_SOCKET_DIR` to a short absolute directory under private root-owned ancestry, create it with mode `0700`, and keep it owned by the Pi user. The extension validates that directory and forwards it as upstream `AGENT_BROWSER_SOCKET_DIR`; ambient upstream socket overrides remain ignored.
|
|
217
|
-
|
|
218
|
-
In a Linux user namespace, `/` may report an unmapped owner. Socket checks trust that operating-environment root while retaining checks on every other ancestor and the private socket directory; they do not protect against whoever controls the root filesystem. The demonstrated bubblewrap layout uses a read-only mode-`0755` root and a current-user-owned mode-`0700` `/tmp`, with private HOME and checkout below it. Unmapped non-root ancestors such as `/home` still fail automatic restore checks; this is not support for every bubblewrap layout. See [filesystem trust](docs/ARCHITECTURE.md#ownership).
|
|
219
|
-
|
|
220
|
-
## Optional package config and web search
|
|
221
|
-
|
|
222
|
-
`pi-agent-browser-native` also reads package-owned config under Pi-scoped paths:
|
|
223
|
-
|
|
224
|
-
- global user config: `~/.pi/config/pi-agent-browser-native/config.json`
|
|
225
|
-
- project config: `.pi/config/pi-agent-browser-native/config.json`
|
|
226
|
-
- explicit override: `PI_AGENT_BROWSER_CONFIG=/path/to/config.json`
|
|
227
|
-
|
|
228
|
-
`pi install npm:pi-agent-browser-native` loads the extension, but it does **not** usually put the package helper on your shell `PATH`. You can configure web search by writing the config file directly, or run the helper through `npm exec` when you want a command to write it for you.
|
|
229
|
-
|
|
230
|
-
Inspect paths/status with the helper when available on `PATH`, or through npm:
|
|
231
|
-
|
|
232
|
-
```bash
|
|
233
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config paths
|
|
234
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config show
|
|
235
|
-
```
|
|
236
|
-
|
|
237
|
-
The optional `agent_browser_web_search` companion tool is available when a usable Exa or Brave credential source is configured or resolvable from startup config or trusted session config. It is not an `agent_browser` input mode and does not launch a browser; prefer it for current/live external web facts and URL discovery, then use `agent_browser` when the page itself needs interaction, screenshots, authenticated/profile content, or DOM inspection. Prefer it over automating public search-engine forms such as Google in headless browser jobs: those flows may be redirected to anti-bot or CAPTCHA pages, and this wrapper does not provide or recommend CAPTCHA bypass. If both keys are available, the default provider is Exa because its `/search` endpoint returns agent-friendly highlights and search modes; set `webSearch.preferredProvider` to `"brave"` when you prefer Brave Search.
|
|
238
|
-
|
|
239
|
-
Get an Exa API key from the [Exa dashboard](https://dashboard.exa.ai/api-keys) or a Brave Search API key from the [Brave Search API dashboard](https://api-dashboard.search.brave.com/). Most users can simply export `EXA_API_KEY` or `BRAVE_API_KEY` in the environment that launches `pi`; config is only needed when you want Pi-scoped secret references, a preferred provider, a default Exa search type, or to disable this built-in search tool.
|
|
240
|
-
|
|
241
|
-
Most config users should store env-var references in the Pi-scoped config:
|
|
242
|
-
|
|
243
|
-
```bash
|
|
244
|
-
mkdir -p ~/.pi/config/pi-agent-browser-native
|
|
245
|
-
cat > ~/.pi/config/pi-agent-browser-native/config.json <<'JSON'
|
|
246
|
-
{
|
|
247
|
-
"version": 1,
|
|
248
|
-
"webSearch": {
|
|
249
|
-
"enabled": true,
|
|
250
|
-
"preferredProvider": "exa",
|
|
251
|
-
"defaultSearchType": "deep-lite",
|
|
252
|
-
"exaApiKey": "$EXA_API_KEY",
|
|
253
|
-
"braveApiKey": "$BRAVE_API_KEY"
|
|
254
|
-
}
|
|
255
|
-
}
|
|
256
|
-
JSON
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
`pi install` does not add package helper binaries to your shell `PATH`. Use direct JSON config edits, or run the helper only through `npm exec`:
|
|
260
|
-
|
|
261
|
-
```bash
|
|
262
|
-
# Store env-var references in global config.
|
|
263
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search set-env EXA_API_KEY --global
|
|
264
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search set-env BRAVE_API_KEY --global
|
|
265
|
-
|
|
266
|
-
# Store an env-var reference in project config.
|
|
267
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search set-env EXA_API_KEY --project
|
|
268
|
-
|
|
269
|
-
# Prefer Brave when both Exa and Brave keys are available, or clear with "auto".
|
|
270
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search prefer brave --global
|
|
271
|
-
|
|
272
|
-
# Disable this package's built-in web-search tool in global config even if API keys are in the environment.
|
|
273
|
-
# Global disable applies to normal runs unless a project config or PI_AGENT_BROWSER_CONFIG override explicitly re-enables it.
|
|
274
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search disable --global
|
|
275
|
-
|
|
276
|
-
# Hard-disable web search for one run, regardless of project config, by using the highest-priority override layer.
|
|
277
|
-
cat > /tmp/pi-agent-browser-disable-web-search.json <<'JSON'
|
|
278
|
-
{ "version": 1, "webSearch": { "enabled": false } }
|
|
279
|
-
JSON
|
|
280
|
-
PI_AGENT_BROWSER_CONFIG=/tmp/pi-agent-browser-disable-web-search.json pi
|
|
281
|
-
|
|
282
|
-
# Store a plaintext key in Pi-scoped user config; output stays redacted.
|
|
283
|
-
printf '%s' "$EXA_API_KEY" | npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search set-key --provider exa --stdin
|
|
284
|
-
|
|
285
|
-
# Store a secret-manager command source. Add --project when you want the repo config to own the source.
|
|
286
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config web-search set-command "op read 'op://Private/Brave Search/API Key'" --provider brave --global
|
|
287
|
-
```
|
|
288
|
-
|
|
289
|
-
Config merges in this order: global → project → `PI_AGENT_BROWSER_CONFIG` override. The globally installed or CLI-loaded extension still loads project-local `.pi/config/pi-agent-browser-native/config.json` when Pi trust allows that project layer; it skips that project layer when Pi reports the project is untrusted or when Pi is launched with `--no-approve`. `webSearch.enabled` is evaluated after the loaded layers merge. Use `web-search disable --global` for a user default, `web-search disable --project` for one repo, and a `PI_AGENT_BROWSER_CONFIG` override with `{ "webSearch": { "enabled": false } }` when web search must stay off even if project config exists. Loaded config may use plaintext, custom environment aliases, interpolation literals, malformed-or-late-bound `$` values, and `!command` credential sources; the resolved secret is passed to the provider request while tool content, details, status output, and docs examples stay redacted. `web-search set-key`, `set-command`, and `clear` require `--provider`; `set-env` infers Exa/Brave from `EXA_API_KEY` or `BRAVE_API_KEY` unless you pass `--provider`.
|
|
290
|
-
|
|
291
|
-
For Exa, the effective mode is the per-call `searchType`, then `webSearch.defaultSearchType`, then `auto`. A research-heavy coding workflow should set the config default to `deep-lite`; callers can still override it per search. Users who do not opt in keep the existing `auto` latency.
|
|
292
|
-
|
|
293
|
-
| Exa `searchType` | Typical latency | Use |
|
|
294
|
-
| ---------------- | --------------- | --------------------------------------------- |
|
|
295
|
-
| `instant` | ~250 ms | Trivial lookups only |
|
|
296
|
-
| `fast` | ~450 ms | Low-latency relevance |
|
|
297
|
-
| `auto` | ~1 s | Everyday fact lookup |
|
|
298
|
-
| `deep-lite` | ~4 s | Preferred research-before-implementation mode |
|
|
299
|
-
| `deep` | 4–15 s | Hard multi-source research and comparisons |
|
|
300
|
-
| `deep-reasoning` | 12–40 s | Exhaustive or hardest multi-hop research only |
|
|
301
|
-
|
|
302
|
-
```json
|
|
303
|
-
{
|
|
304
|
-
"query": "pi-agent-browser-native agent_browser_web_search searchType defaults",
|
|
305
|
-
"searchType": "deep-lite",
|
|
306
|
-
"count": 5
|
|
307
|
-
}
|
|
308
|
-
```
|
|
309
|
-
|
|
310
|
-
Exa calls may also use up to 20 `includeDomains` or `excludeDomains`, a typed `category`, up to 10 deep-mode `additionalQueries`, and the `highlightsDynamic` research preview. `company` and `people` categories cannot combine with `freshness` or `excludeDomains`. These explicit Exa-only options fail clearly when Brave is selected; the existing `searchType` field remains ignored by Brave. Regular `contents.highlights: true` stays the default, and structured output schemas remain out of scope.
|
|
311
|
-
|
|
312
|
-
Every Exa request asks the provider to prefer primary official sources, honor requested versions/dates, and avoid equivalent results. After provider normalization, both adapters remove later results with the same exact normalized URL while preserving first-result order; they do not guess that distinct paths or query URLs are aliases and do not overfetch to replace removed rows. `details.duplicatesRemoved` reports any shrinkage. Exa `publishedDate` and Brave `page_age` appear as `pageDate`; Brave can also return a separate result `age`. These are provider-supplied page clues, not crawl age or proof of a version match. For version-sensitive work, inspect those clues, constrain one follow-up to the primary domain (`includeDomains` for Exa or `site:` in a Brave query), then read the primary page.
|
|
313
|
-
|
|
314
|
-
The same config file can record conservative browser defaults such as a profile hint or a Chromium-compatible executable path:
|
|
315
|
-
|
|
316
|
-
```bash
|
|
317
|
-
# Ask the agent to use this profile for signed-in/account-specific work.
|
|
318
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config browser profile set "Profile 1" --policy authenticated-only
|
|
319
|
-
|
|
320
|
-
# Ask the agent to launch a different Chromium-compatible browser executable.
|
|
321
|
-
npm exec --yes --package pi-agent-browser-native@latest -- pi-agent-browser-config browser executable set "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser"
|
|
322
|
-
```
|
|
323
|
-
|
|
324
|
-
Profile hints with `authenticated-only` remain advisory. Global/override profile names with `policy: "always"` and executable defaults bootstrap inactive automatic root browsers only when the effective engine is Chrome; active roots and unrelated explicit sessions retain their native settings. Configure profile/executable guidance globally, in trusted project config, or through `PI_AGENT_BROWSER_CONFIG`. Ask the agent to run `agent_browser` with `args: ["profiles"]` and `args: ["doctor"]` when profile resolution fails. The upstream `profiles` command lists Chrome profiles from Chrome's user data directory; `Default` is not canonical on every machine. Use the displayed profile directory name, a full profile/user-data directory path when upstream accepts one, or a configured `browser.executablePath` plus `sessionMode: "fresh"` for a different Chromium-compatible browser.
|
|
325
|
-
|
|
326
|
-
## Common agent calls
|
|
327
|
-
|
|
328
|
-
You usually prompt the agent in natural language. These JSON snippets show the exact native tool shape the agent should use.
|
|
329
|
-
|
|
330
|
-
Open a page and inspect it (first-call recipe: open → snapshot -i → interact with current `@refs` → snapshot -i after changes). Omit `--json` unless you need JSON text; structured details are available by default. Explicit `--json false` selects native text without trusted row, confirmation, lifecycle, page-target or artifact receipts; see [the output contract](docs/TOOL_CONTRACT.md#wrapper-json).
|
|
46
|
+
For example, Pi opens and inspects a page with these calls:
|
|
331
47
|
|
|
332
48
|
```json
|
|
333
49
|
{ "args": ["open", "https://example.com"] }
|
|
334
50
|
{ "args": ["snapshot", "-i"] }
|
|
335
51
|
```
|
|
336
52
|
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
Local Chrome startup uses stock `--args --no-startup-window` to suppress Chrome's extra startup window when native environment/config does not already own local launch defaults. In that case, caller CLI arguments follow the startup default unchanged. With native environment/config launch defaults, the wrapper injects no Chrome startup arguments; include `--no-startup-window` in your native `args` when a single headed window is wanted. Caller settings retain native CLI → environment → config precedence. Headless remains the default, ordinary follow-ups do not resend wrapper bootstrap settings, and changing native launch settings can restart a browser. CDP, providers, Electron, and Lightpanda remain native-owned; no tabs are deleted or profiles edited.
|
|
340
|
-
|
|
341
|
-
### Opt-in Lightpanda
|
|
342
|
-
|
|
343
|
-
Chrome remains the default. Install [Lightpanda](https://github.com/lightpanda-io/browser/releases/tag/1.0.0) separately on supported macOS/Linux hosts, put `lightpanda` on `PATH`, then select it explicitly:
|
|
344
|
-
|
|
345
|
-
```json
|
|
346
|
-
{ "args": ["--engine", "lightpanda", "open", "https://example.com"], "sessionMode": "fresh" }
|
|
347
|
-
{ "args": ["snapshot", "-i"] }
|
|
348
|
-
```
|
|
349
|
-
|
|
350
|
-
`--engine` is launch-scoped; changing an active managed browser uses the existing fresh-session recovery. `AGENT_BROWSER_ENGINE` and native config also select the engine. Wrapper-supplied Chrome profile/executable defaults are omitted for Lightpanda; caller arguments, environment, and native config remain unchanged.
|
|
351
|
-
|
|
352
|
-
Lightpanda screenshots render page text, not graphical layout. Positively identified captures carry `imageObservations[].rendering: "text"` and unknown coordinate geometry; keep using refs, not image-derived mouse coordinates. Lightpanda is not Chrome-parity visual QA, profile login, multi-tab, or WebMCP support, and this wrapper never falls back automatically. See [capabilities and limits](docs/COMMAND_REFERENCE.md#lightpanda-engine).
|
|
353
|
-
|
|
354
|
-
### Native navigation and interaction
|
|
355
|
-
|
|
356
|
-
`open` without a URL uses native `get url`: it launches when needed and otherwise preserves the current page. This also applies to effective batch rows and code calls. `details.args` preserves the request, while `effectiveArgs`, command, URL data, and lifecycle describe the actual native operation. Use `open about:blank` only when you intend to navigate there. Help/version requests remain unchanged.
|
|
357
|
-
|
|
358
|
-
Watch a browser window during a demo, QA run, or user-completed login by adding upstream's global `--headed` flag on the first launch. Use `sessionMode: "fresh"` if a managed session may already exist, because headed/headless state is launch-scoped. A successful first/fresh local wrapper-managed headed launch, including a launch inside `batch`, returns `details.browserWindow = { mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` and one visible handoff sentence; CDP, auto-connect, provider, and Electron attachments do not. This proves that the wrapper requested and upstream launched headed mode, not that the OS window is visible on the user's display; remote, container, or virtual-display setups can still hide it. After the user finishes in the window, continue with `sessionMode: "auto"`.
|
|
359
|
-
|
|
360
|
-
```json
|
|
361
|
-
{ "args": ["--headed", "open", "https://example.com"], "sessionMode": "fresh" }
|
|
362
|
-
{ "args": ["screenshot", "/tmp/agent-browser-headed-check.png"] }
|
|
363
|
-
```
|
|
364
|
-
|
|
365
|
-
For wrapper-owned headed launches, the extension disables upstream 0.33.2 periodic restore autosave by default because its multi-origin collector opens visible temporary tabs and can delay daemon policy inspection. The extension records the effective launch-time interval and reapplies it to every helper and follow-up subprocess, including still-owned off-current sessions (also after failed replacement cleanup), Electron cleanup closes, and reload/resume, so the receiving daemon does not see changing configuration. Native `close` still saves, but upstream exempts headed browsers from idle shutdown, so closing the window by hand can lose newer state. Set `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` before launch when periodic preservation matters; changing it on a running wrapper-owned headed daemon is rejected until you close that session and launch fresh.
|
|
366
|
-
|
|
367
|
-
Render a WebGPU page by enabling upstream's WebGPU launch preset on a fresh local browser:
|
|
368
|
-
|
|
369
|
-
```json
|
|
370
|
-
{ "args": ["--webgpu", "open", "https://webgpu.github.io/webgpu-samples/?sample=helloTriangle"], "sessionMode": "fresh" }
|
|
371
|
-
{ "args": ["screenshot", "/tmp/webgpu.png"] }
|
|
372
|
-
```
|
|
373
|
-
|
|
374
|
-
`--webgpu` is also available as `AGENT_BROWSER_WEBGPU`; `--webgpu false` overrides an enabled environment default. Standalone upstream also supports `"webgpu": true` in `agent-browser.json`; native calls preserve upstream config. It cannot be combined while enabled with `--cdp`, `--auto-connect`, or provider launches. Run `{ "args": ["doctor", "--webgpu"] }` to pixel-check rendering and capture. macOS supports headless WebGPU screenshots; upstream requires a logged-in headed desktop on Windows and `--headed` plus Vulkan loader/Mesa packages on Linux (automatic Xvfb unless `AGENT_BROWSER_NO_XVFB=1`).
|
|
375
|
-
|
|
376
|
-
On `agent-browser 0.35.0`, trust a private interception-proxy CA for locally launched Linux Chromium with a fresh, restore-disabled session:
|
|
377
|
-
|
|
378
|
-
```json
|
|
379
|
-
{
|
|
380
|
-
"args": [
|
|
381
|
-
"--proxy",
|
|
382
|
-
"http://proxy.example:8080",
|
|
383
|
-
"--ca-cert",
|
|
384
|
-
"/path/to/proxy-ca.pem",
|
|
385
|
-
"open",
|
|
386
|
-
"https://example.com"
|
|
387
|
-
],
|
|
388
|
-
"sessionMode": "fresh"
|
|
389
|
-
}
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
`--ca-cert <path>` also has `AGENT_BROWSER_CA_CERT`; `--no-ca-cert` / `AGENT_BROWSER_CLEAR_CA_CERT` clears retained trust. Upstream accepts PEM bundles or DER certificates, uses an isolated NSS store, preserves normal hostname/validity checks, and requires Linux Chromium plus `certutil`. It rejects profiles, CDP/auto-connect, providers, Lightpanda, `--ignore-https-errors`, and non-Linux hosts. Because a trusted interception CA can observe authenticated traffic, this wrapper disables automatic managed restore for CA-enabled sessions.
|
|
393
|
-
|
|
394
|
-
Restrict browser and `read` traffic with upstream's domain containment on a fresh local Chrome context:
|
|
395
|
-
|
|
396
|
-
```json
|
|
397
|
-
{
|
|
398
|
-
"args": ["--allowed-domains", "example.com,*.example.org", "open", "https://example.com"],
|
|
399
|
-
"sessionMode": "fresh"
|
|
400
|
-
}
|
|
401
|
-
```
|
|
402
|
-
|
|
403
|
-
In `agent-browser 0.32.0`, the allowlist also covers workers and popups and disables Chromium `RTCPeerConnection` while active. Upstream owns containment and rejects incompatible CDP/auto-connect, profile, restore/state, provider, iOS/Safari, and startup-argument combinations; the wrapper passes the setting and upstream result through unchanged.
|
|
404
|
-
|
|
405
|
-
On `https://example.com/`, the main link label is **Learn more**—use exact visible text from your snapshot, not guessed copy such as `More information...`.
|
|
406
|
-
|
|
407
|
-
Click a visible ref, then refresh refs after navigation or a DOM update:
|
|
408
|
-
|
|
409
|
-
```json
|
|
410
|
-
{ "args": ["click", "@e2"] }
|
|
411
|
-
{ "args": ["snapshot", "-i"] }
|
|
412
|
-
```
|
|
413
|
-
|
|
414
|
-
Run a multi-step flow in one tool call:
|
|
415
|
-
|
|
416
|
-
```json
|
|
417
|
-
{
|
|
418
|
-
"args": ["batch", "--bail"],
|
|
419
|
-
"stdin": "[[\"open\",\"https://example.com\"],[\"snapshot\",\"-i\"]]"
|
|
420
|
-
}
|
|
421
|
-
```
|
|
422
|
-
|
|
423
|
-
Use exact `batch --bail` when a later content step assumes an earlier navigation succeeded. Without fail-fast behavior, a failed navigation can leave an unverified prior page active; the wrapper rejects that shape before the content step. Non-bail continuation remains available when every possible retained target is verified. Splitting navigation and content into separate calls is the other safe option.
|
|
53
|
+
Snapshots give Pi current `@eN` references for controls. Refresh the snapshot after navigation or page changes. Check the resulting page after a click; command success alone does not prove the app changed.
|
|
424
54
|
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
Read documentation or other unstructured text without requiring a Chrome page, or omit the URL to read the rendered DOM of the current tab:
|
|
428
|
-
|
|
429
|
-
```json
|
|
430
|
-
{ "args": ["read", "https://example.com/docs", "--filter", "authentication"] }
|
|
431
|
-
{ "args": ["read"] }
|
|
432
|
-
```
|
|
433
|
-
|
|
434
|
-
Explicit URL reads prefer `text/markdown`, then try a `.md` path and nearby `llms.txt` links before falling back to readable HTML text. Use `--outline`, `--llms index|full`, `--require-md`, `--raw`, or `--timeout <ms>` when needed. Explicit reads, including all-read batches, do not allocate or replace a managed browser, require a known page, or run browser helpers before, after, or following a timeout. Native config and argv pass through unchanged; the native HTTP reader owns fetching. A visible `Read execution` line reports the source, CLI start and native launch evidence without treating an HTTP read as proof of shared-browser liveness. Bare `read` still uses and verifies the current DOM. The native no-browser-effects path requires the companion upstream fix; older supported binaries do not guarantee it merely because this wrapper skips helpers. It renders upstream `data.content` first, preserves metadata in `details.data`, keeps fetched URLs from replacing the active browser tab target, and budgets explicit long read timeouts across upstream's `.md` and ancestor-`llms.txt` request fallbacks.
|
|
435
|
-
|
|
436
|
-
Use `session info` to inspect browser status. A timeout proves no liveness and returns only `retry-session-info` for that same session/namespace; it does not run page probes or change existing page/ref state.
|
|
437
|
-
|
|
438
|
-
Evaluate page JavaScript through stdin. Put the script in the top-level `stdin` field, not as an extra `args` token after `--stdin`. Return the value you want as an expression; `eval --stdin` may warn with `details.evalStdinHint` when a function-shaped snippet serializes to `{}` instead of being invoked:
|
|
439
|
-
|
|
440
|
-
```json
|
|
441
|
-
{ "args": ["eval", "--stdin"], "stdin": "document.title" }
|
|
442
|
-
{ "args": ["eval", "--stdin"], "stdin": "({ title: document.title, url: location.href })" }
|
|
443
|
-
{ "args": ["eval", "--stdin"], "stdin": "({ title: document.title, url: location.href })", "outputPath": "logs/page-state.json" }
|
|
444
|
-
```
|
|
445
|
-
|
|
446
|
-
Native text cookie reads redact `name=value` values, including empty cookie names; storage reads redact `key: value`, including empty and colon-containing keys, using the same benign-value policy as JSON. Standalone `storage local|session get <key>` and native shorthand `storage local|session <key>` protect the entire sensitive value, including embedded newlines, even when invocation display redaction changes the supplied key. Names and benign preferences remain visible in content, details, spills and exports. Native all-entry storage and raw batches do not escape multiline values, so text redaction is heuristic: use structured JSON when sensitive storage must be unambiguous.
|
|
447
|
-
|
|
448
|
-
Use `outputPath` when `eval`, `get`, `snapshot`, or another extraction should be saved as a durable workspace file. Recording results also export on failure or timeout: their JSON envelope retains the failed attempt, native receipt, verification and any recovery evidence rather than writing misleading bare success data. Keep it distinct from screenshot, download, recording, and other browser artifact destinations; preflight rejects known same-call aliases before browser activity, and the result writer preserves the browser artifact if an alias becomes apparent only afterward. The wrapper writes `details.data` when present, otherwise the model-facing text content. When presentation compacted a large direct result, a result row, or the whole `batch`, it instead reads the full command-redacted pre-compaction payload only from the corresponding live wrapper-managed spill recorded in `details.artifactManifest`; if any required spill is unavailable or untrusted, the call fails without writing compact metadata to the requested path. `details.outputFile` reports the saved path and byte count. Explicit upstream `--json` content stays parseable, including early snapshot/network filters, scroll results, and preparation failures; metadata stays in `details` and the save notice lives only in `details.outputFile`. Help/version output remains native text.
|
|
449
|
-
|
|
450
|
-
Recording receipts separate captured-frame rate and wall-clock capture duration from nominal/output FPS. `details.artifacts[].recording` includes native capture timestamps, received frame counts (not pixel-unique frames), encoded/written/held/dropped/skipped counts, and output duration. Missing native metrics stay unknown. Repeated, static, sparse or final-state-only frames cannot establish UI smoothness.
|
|
451
|
-
|
|
452
|
-
```json
|
|
453
|
-
{ "args": ["record", "stop"], "outputPath": "logs/record-stop.json" }
|
|
454
|
-
```
|
|
455
|
-
|
|
456
|
-
A timed-out stop or `No recording in progress` response gets one bounded native `session info` query. Only a matching terminal native receipt, successful encoder measurements and a verified file can recover success; file presence alone cannot. The original attempt remains visible. A receipt can verify the recording from a timed-out batch without proving its other steps succeeded. Follow the exact status/stop actions returned, not blind retries or longer timeouts. See [recording receipts](docs/TOOL_CONTRACT.md#recording-receipts-and-recovery) for fields and native availability.
|
|
457
|
-
|
|
458
|
-
Extract several known refs or selectors in one `batch` call instead of many serial getter calls. When a prior snapshot and session are available and the same-page freshness checks apply, ref-consuming calls add one extra `snapshot -i` preflight per top-level call or batch. Batching shares that probe across rows; it does not remove it:
|
|
459
|
-
|
|
460
|
-
```json
|
|
461
|
-
{ "args": ["batch"], "stdin": "[[\"get\",\"text\",\"@e64\"],[\"get\",\"text\",\"@e65\"]]" }
|
|
462
|
-
```
|
|
463
|
-
|
|
464
|
-
Save an auth profile without putting the password in `args`:
|
|
465
|
-
|
|
466
|
-
```json
|
|
467
|
-
{ "args": ["auth", "save", "demo", "--password-stdin"], "stdin": "<password>" }
|
|
468
|
-
```
|
|
469
|
-
|
|
470
|
-
Download a file from a known link or control:
|
|
471
|
-
|
|
472
|
-
```json
|
|
473
|
-
{ "args": ["download", "@e5", "/tmp/report.pdf"] }
|
|
474
|
-
```
|
|
475
|
-
|
|
476
|
-
### Persistent browser code
|
|
477
|
-
|
|
478
|
-
Use `agent_browser_code` for loops, branching, or aggregation. Each call gets fresh JavaScript variables and the same browser selected by ordinary `agent_browser` calls. Cookies, open tabs, and authenticated browser state persist according to native session/profile settings; finishing a code call does not close the browser.
|
|
479
|
-
|
|
480
|
-
```json
|
|
481
|
-
{
|
|
482
|
-
"code": "const titles = []; for (const url of ['https://example.com', 'https://example.org']) { const opened = await browser({ args: ['open', url] }); if (!opened.success) throw new Error(opened.error); const title = await browser({ args: ['get', 'title'] }); if (!title.success) throw new Error(title.error); titles.push({ url, title: title.data.title ?? title.data.result }); } emit(titles);"
|
|
483
|
-
}
|
|
484
|
-
```
|
|
485
|
-
|
|
486
|
-
`await browser({ args, stdin?, timeoutMs? })` returns the canonical observation: `success`, `data`, error/category information, exact `nextActions`, and artifact/image evidence when available. Emit the selected JSON the model needs instead of entire observations. `emitImage(observation.imageObservations[0])` selects a verified image from this code call for actual image attachment; emitting a path as JSON does not attach it. Stop at observations that need model judgment rather than guessing the next action in code.
|
|
487
|
-
|
|
488
|
-
```json
|
|
489
|
-
{
|
|
490
|
-
"code": "const shot = await browser({ args: ['screenshot', '/tmp/current-page.png'] }); if (!shot.success) throw new Error(shot.error); emit(shot.artifactVerification); if (shot.imageObservations?.length) emitImage(shot.imageObservations[0]);"
|
|
491
|
-
}
|
|
492
|
-
```
|
|
493
|
-
|
|
494
|
-
Optional `session` and `namespace` select one native identity for the whole code call; omitted fields follow normal browser defaults. Inner native commands can use batch, auth, profiles, state, connect, and close, but cannot change that identity or issue namespace-wide `close --all`. Use the direct tool for those operations. Parent-side native commands keep normal auth/config/environment behavior. The fresh permissioned Node child exposes no host filesystem, network, process APIs, imports, timers, or dynamic code generation.
|
|
495
|
-
|
|
496
|
-
Limits: 25 attempted browser calls; 64 KiB source and emitted JSON; 1 MiB per IPC message and 8 MiB cumulative IPC; 120 seconds by default, 300 seconds maximum. Inner calls are serial, even with `Promise.all`. Selected images are limited to eight and 20 MiB total, with the normal configurable inline limit (5 MiB per image by default). See the [code contract](docs/TOOL_CONTRACT.md#agent_browser_code) for failure, cancellation, and image rules.
|
|
497
|
-
|
|
498
|
-
A handled browser failure remains visible in `failures` and `codeRun.failedCallCount`; successful source completion is not proof that every inner command succeeded. Rejected inner input shapes, source exceptions, cancellation, or deadline failure fail the code result. Uncertain mutations must be inspected before retry. Ordered Pi transition entries preserve inner page/ref/attachment/close state through resume; an interrupted intent remains unknown until inspection. A persisted Pi session is required.
|
|
499
|
-
|
|
500
|
-
### Advanced browser tools
|
|
501
|
-
|
|
502
|
-
List specialized capabilities with `agent_browser_tools {}`. Enable only what the task needs:
|
|
503
|
-
|
|
504
|
-
```json
|
|
505
|
-
{ "enable": ["action", "qa", "electron", "source", "network"] }
|
|
506
|
-
```
|
|
507
|
-
|
|
508
|
-
Activation adds tools without removing other active tools; Pi owns the selected-tool history. Omit CLI `--tools` for normal lazy activation, or include each desired advanced tool in that explicit selection. The loader cannot enable tools excluded by the host. Each specialized tool takes **flat fields**, not a nested object on `agent_browser`:
|
|
509
|
-
|
|
510
|
-
| Tool | Example input | Preserved capability |
|
|
511
|
-
| ------------------------------ | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
512
|
-
| `agent_browser_action` | `{ "action": "fill", "locator": "label", "value": "Email", "text": "user@example.com" }` | Stable locators, direct selectors/refs, and unique accessible native dropdown selection |
|
|
513
|
-
| `agent_browser_qa` | `{ "url": "https://example.com", "expectedText": "Example Domain" }` | Visible assertions, scoped diagnostics, actionable/benign error classification, and a real pass/fail verdict |
|
|
514
|
-
| `agent_browser_electron` | `{ "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" }` | Host discovery, isolated app launch, CDP readiness, status/probe, and owned cleanup |
|
|
515
|
-
| `agent_browser_source` | `{ "selector": "#save", "componentName": "SaveButton" }` | Experimental DOM/React and bounded local workspace candidates |
|
|
516
|
-
| `agent_browser_network_source` | `{ "requestId": "req-1", "url": "/api/fail" }` | Experimental failed-request/initiator/local-source correlation |
|
|
517
|
-
|
|
518
|
-
Use `agent_browser_action` with `action: "select"` for a native dropdown: a direct `selector` plus `value`/`values`, or an exact role/name or label locator resolved to one visible combobox/listbox. For framework-controlled rich inputs, inspect returned recovery actions, focus a current ref, then use native `keyboard type`; verify application state before submitting.
|
|
519
|
-
|
|
520
|
-
QA defaults to `domcontentloaded`. URL QA clears enabled network/console buffers and checks page-error residue before opening the URL. Matched residue makes that check unverified and non-pass; it is not proof of a new application error. Attached QA (`{ "attached": true }`) preserves existing buffers and defaults diagnostics off; opt into `checkNetwork`, `checkConsole`, or `checkErrors` deliberately. Native diagnostic-read success alone is not a QA verdict.
|
|
521
|
-
|
|
522
|
-
Electron launch uses a new temporary app profile and does not inherit the normal app's signed-in state. Use `agent_browser_electron` to list, launch, probe/status, then clean up its returned `launchId`. A manually launched debug-enabled app uses native `connect` and stays host-owned. Read [the Electron guide](docs/ELECTRON.md) for lifecycle and timeout details.
|
|
523
|
-
|
|
524
|
-
Source tools report candidates, never authoritative blame or automatic edits. Workspace scans default to 2,000 source files, cap at 5,000, and return at most ten workspace matches. They remain available even when unrelated Pi file tools are disabled; packaged app bundles are not unpacked.
|
|
525
|
-
|
|
526
|
-
### Fixed sequences and evidence
|
|
527
|
-
|
|
528
|
-
Use native `batch --bail` for known sequences. Add an explicit destination/text wait after a navigation-prone click; successful dispatch is not proof the application changed. Split when fresh refs or model judgment are needed.
|
|
529
|
-
|
|
530
|
-
```json
|
|
531
|
-
{
|
|
532
|
-
"args": ["batch", "--bail"],
|
|
533
|
-
"stdin": "[[\"open\",\"https://shop.example/checkout\"],[\"fill\",\"#email\",\"user@example.com\"],[\"click\",\"#continue\"],[\"wait\",\"--url\",\"**/shipping\"],[\"wait\",\"--text\",\"Shipping address\"],[\"screenshot\",\"/tmp/shipping.png\"]]"
|
|
534
|
-
}
|
|
535
|
-
```
|
|
536
|
-
|
|
537
|
-
Use the exact returned recovery payload after errors. Timeout partial progress reports current page and file observations separately from planned steps, whose outcomes remain unknown without upstream results. An existing artifact or matching URL does not prove any step ran. Multi-step timeouts offer inspection rather than guessing a retry position; a single retryable step may offer a retry candidate after inspection. For paced typing, native batch can interleave `keyboard type` characters and positional `wait <ms>` rows; see [the migration map](docs/TOOL_CONTRACT.md#07-migration). Do not assume a native `--delay` option reproduces the removed job behavior.
|
|
538
|
-
|
|
539
|
-
Both JSON and prose results expose a bounded canonical observation with recovery actions and artifact receipts. Oversized observations name an `observationPath` for the complete redacted result; audit-only `details` is not the model's sole recovery channel. Inner code calls avoid full model rendering, spills, and image encoding until output is selected.
|
|
540
|
-
|
|
541
|
-
Screenshots expose measured image dimensions and capture geometry when supported by matching before/after samples. A full-page or element image is not a viewport screenshot. Native mouse coordinates are viewport CSS pixels; use the [geometry contract](docs/TOOL_CONTRACT.md#image-observations) and current scroll/frame state before converting image pixels. Unknown geometry stays unknown, including ambiguous or scrolled element captures, unmeasured batch captures, and Lightpanda text-rendered images (`rendering: "text"`). Pi image-resize notes only map sent pixels to original image pixels.
|
|
542
|
-
|
|
543
|
-
Save artifacts to the user's exact paths and check `artifactVerification` before claiming they exist or are fresh. Conditional `changed: false` screenshots produce no new attachment. `outputPath` is a separate result-data destination and must not alias a screenshot, download, or recording. Close ends the browser session; explicit files remain host-owned. Recording starts remain pending until stop and receipt/file verification.
|
|
544
|
-
|
|
545
|
-
### Upgrading from 0.6
|
|
546
|
-
|
|
547
|
-
Version 0.7 replaces top-level `script`/`job` and nested advanced input modes. Move script bodies to `agent_browser_code.code`, replace `.ok` with `.success`, and account for the now-persistent browser. Move fixed jobs to native batch; enable and call the flat advanced tools above. There is no parallel legacy public execution route. See [the complete migration map](docs/TOOL_CONTRACT.md#07-migration) and [package rollback](docs/RELEASE.md#07-upgrade-and-rollback).
|
|
548
|
-
|
|
549
|
-
Restart every participating Pi process onto the same updated package before relying on cross-process coordination. Package rollback means reinstalling the previously used 0.6 release and restarting those processes, preserving saved authentication, artifacts, and Pi transcripts; it does not undo browser actions.
|
|
550
|
-
|
|
551
|
-
### Working-directory changes
|
|
552
|
-
|
|
553
|
-
With `pi-change-working-dir`, each browser call captures the selected execution directory once. New relative screenshots, downloads, recordings, file inputs, `outputPath`, and source/network-source scans use that directory, including queued batches and code calls. Absolute paths remain unchanged. Existing browsers, tabs, profiles, restore identities, pending recording destinations, cached artifacts, and Pi package config/trust remain anchored to their original owners.
|
|
554
|
-
|
|
555
|
-
Ordinary native config discovery stays at the project or current managed browser's launch directory. `--session` selects a target for that call; it does not switch the config root. An unnamed `sessionMode: "fresh"` launch and explicit `--config` / `AGENT_BROWSER_CONFIG` resolve from the selected execution directory, preserving native session precedence. Managed fresh launches retain their launch directory across follow-ups and resume. If the old launch directory is deleted, ordinary browser calls stop with recovery guidance instead of selecting another project's config; restore it or explicitly choose fresh/config from the new directory. Shell/file operations can continue independently. Older directory extensions must be updated and Pi restarted; without a directory extension, native Pi cwd remains the default.
|
|
556
|
-
|
|
557
|
-
## Authenticated/profile workflows
|
|
558
|
-
|
|
559
|
-
### Shared browser defaults
|
|
560
|
-
|
|
561
|
-
Ordinary browser calls use one native named browser per **root Pi session**. A parent and its subagents share that browser; unrelated roots get different names and can browse concurrently. `pi-subagents` supplies `PI_SUBAGENT_ROOT_SESSION_ID` through its existing launch records, including detached runs, delegated forks, grandchildren, and revived children. Ordinary Pi new/fork/clone sessions get their own root identity; resuming the same root or changing its cwd keeps its browser name.
|
|
562
|
-
|
|
563
|
-
Cooperating updated Pi processes share a cross-process lock keyed by the actual socket context, canonical namespace, and session. It covers a direct call's helpers and action, or an entire code call; different identities can run independently. Coordinate navigation across separate tool calls with existing subagent/intercom tools. This is ordering, not rollback or protection against human navigation, external CLI clients, or older extension versions. Parent or child exit does not close the group browser. Native idle policy applies; explicitly close only your group's browser when the group is finished.
|
|
564
|
-
|
|
565
|
-
To bootstrap new root browsers from an authenticated normal Chrome profile, set the existing **Pi package** config in `~/.pi/config/pi-agent-browser-native/config.json`:
|
|
566
|
-
|
|
567
|
-
```json
|
|
568
|
-
{
|
|
569
|
-
"browser": {
|
|
570
|
-
"defaultProfile": { "name": "Default", "policy": "always" },
|
|
571
|
-
"executablePath": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
|
|
572
|
-
}
|
|
573
|
-
}
|
|
574
|
-
```
|
|
575
|
-
|
|
576
|
-
For inactive automatic roots, global/override `always` profile **names** and the configured executable bootstrap native launch through a scoped subprocess environment shared with helpers. Active roots retain native launch settings instead of resending defaults that could replace an explicitly profiled browser; their existing restore key is preserved, including follow-ups naming the same root browser. Native Chrome profile names use independent temporary copies, not the source profile's lock. Profile paths and other profile policies remain advisory; project guidance does not change a root's launch settings when a child changes cwd. Explicit unrelated sessions and Electron launches do not receive root bootstrap defaults. Code calls selecting the ordinary root browser use its normal configuration and authenticated state. Do not set a global native profile merely to enable this feature: it can change the launch configuration of existing unrelated browsers.
|
|
577
|
-
|
|
578
|
-
Each automatic root has its own stable native restore key. Native save/restore preserves cookies, localStorage, and sessionStorage across normal close/restart; it does not merge auth between roots or save new IndexedDB-only credentials, service workers, or page-memory grants. Named profile copies can bootstrap existing IndexedDB, but are discarded on close and never write back to the communal profile. Verify the actual application after restart, not just a successful restore receipt. If an application needs unsupported state, use a supported persistent **per-root** profile and collect required sign-ins together; never point concurrent roots at one writable profile directory. Sites can expire sessions or require a fresh human challenge.
|
|
579
|
-
|
|
580
|
-
Explicit native `session` defaults (config or `AGENT_BROWSER_SESSION`) still override automatic root selection, including `sessionMode: "fresh"`. They intentionally share whichever browser the caller selected. Per-call session/namespace flags retain precedence. Ordinary launch options such as `--profile` or `--executable-path` still use the root-group name. Repeat explicit options when deliberately relaunching a persistent per-root profile after close. Explicit `fresh` and attachment choices retain their existing managed-session lifecycle; later auto calls follow that selected managed session. Use `session info` with the reported `sessionName` for read-only native daemon/browser evidence. Unknown native fields remain unknown.
|
|
581
|
-
|
|
582
|
-
Set `--confirm-actions` on the session's first command. Its established setting follows same-namespace/session helpers and ordinary, code, advanced and resumed calls, so exact returned confirm/deny actions work without repeating the policy. Bare follow-ups retain a selected string, including explicit `""`, over ambient environment or discovered-config changes. Native unset is not retained and keeps ordinary native default resolution. A new explicit `--confirm-actions` or `--config` selection uses native argv → environment → config precedence and can restart the daemon, losing pending confirmations. Successful close retires the setting. Replacement cleanup uses the old identity's setting and retains its continuity state if close fails or needs confirmation. Electron launch and host helpers use the same selected setting. Definite pre-dispatch, native CLI parse/config and local-inspection rejections do not replace an established setting unless an earlier browser helper admitted it; interrupted work retains its selected intent.
|
|
583
|
-
|
|
584
|
-
Native confirmation controls retain their exact ID, namespace/session and compact command/action provenance in the existing transcript state. Use the exact returned confirm/deny action: matching guarded decisions skip page helpers that could overwrite the native pending slot, then reconcile the completed browser action normally. In native batch, that helper suppression applies only to a matching first effective decision; raw arguments win over stdin and later page/ref checks remain. This does not prove native ID validation or make the action browser-independent. Page content cannot create this provenance. A pending `tab_new` transport may retain paired `after_confirmation: ["open", <requested URL>]` and nonempty `guidance` only when the continuation matches the original tab URL. Its conditional guidance stays visible: approve to create the blank/labeled tab, then explicitly navigate that new tab; denial never authorizes the continuation. The extension does not execute these arguments automatically. Completed confirmed full snapshots install their refs, including in batch row order. A final confirmed capture can supply the next same-identity call's freshness sample once, including semantic role click/check/fill and unique role/label select resolution; intervening calls or later rows consume that opportunity. Pending snapshots render a confirmation control, not an empty page. Target/membership/invalidation and native-generation checks still apply. On this one use, live verification compares the complete capture URL, including fragments, and rejects refs when it changes—including wrapper-owned managed sessions and warm sessions without resume. Ordinary ref/tab URL comparisons remain fragment-insensitive. Only a DOM-only rename on the same page after that sample is not resampled on this one use. Further ref operations use the normal freshness check. Policy-required explicit URL reads keep their separate two-call flow; only `capabilities.readRequiresConfirmation: true` grants browser-independent confirmation with native ID checking. Legacy URL-read prompts retain correct routing but normal page checks. A pending cold-reopen helper exposes its decision without dispatching the requested getter or replaying the original open.
|
|
585
|
-
|
|
586
|
-
Full live identity, browser-independent native reads/confirmations and detailed recording receipts require the companion upstream fixes; the current recommended release does not yet supply all of them. Older supported versions remain accepted with unavailable evidence marked unknown. The extension does not upgrade or restart your installed browser to obtain these fields.
|
|
587
|
-
|
|
588
|
-
### Lifecycle and cleanup
|
|
589
|
-
|
|
590
|
-
Shared root and caller-owned browsers survive Pi quit; managed/fresh browsers and wrapper-launched Electron resources retain ownership-aware cleanup. Close or clean up owned resources explicitly before disabling or removing the extension. Reload after extension removal has no browser handover guarantee. Native save-on-close and per-root restore remain the persistence path; page JavaScript, unsaved forms, and live recordings are not serialized. See [session ownership](docs/ARCHITECTURE.md#session-model).
|
|
591
|
-
|
|
592
|
-
### Profile selection
|
|
593
|
-
|
|
594
|
-
The wrapper does not clone profiles or hide what upstream Chrome/Chromium profile or executable you chose. Passing `--profile` or `--executable-path` is an explicit upstream `agent-browser` choice. Visible page content from real profiles is model-visible and may persist in transcripts or saved artifacts; redaction protects credential-like cookie/storage/auth values, not ordinary page text you asked the browser to read. Redactions use `[REDACTED]` (URL-encoded in parsed URLs); ordinary technical phrases such as `bearer token` stay intact. URL redaction includes `code`, `authorization_session_id`, and auth-context `state` / `nonce` across visible text, details, spills, and `outputPath` exports. URLs needing no redaction keep their original spelling.
|
|
595
|
-
|
|
596
|
-
Use these rules:
|
|
597
|
-
|
|
598
|
-
- Use public/temp profiles for tests and examples.
|
|
599
|
-
- Do not assume `--profile Default` is correct. Ask the agent to run `profiles` to list Chrome profile directory names, then `doctor` if profile/user-data-dir resolution still fails. On macOS, a copied Chrome profile may omit Keychain-encrypted cookies, so profile selection is not proof that the target page is authenticated; verify the page and use a user-approved headed login once when needed.
|
|
600
|
-
- For non-Chrome Chromium browsers such as Brave, Edge, Arc, or Vivaldi, use `--executable-path <path>` when upstream can launch that executable. If you need that browser's existing login state, use the browser's real profile/user-data directory path when upstream accepts it, or attach with `--auto-connect` / `connect` to a debug-enabled running browser when appropriate.
|
|
601
|
-
- Use `sessionMode: "fresh"` when switching from public browsing to `--allowed-domains`, `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--enable`, `-p` / `--provider`, or iOS `--device`.
|
|
602
|
-
- Use `--session` when you want to manage a live upstream session name yourself. For CDP, connect once, verify with `get url`, keep using that session without repeating `--cdp`, and close it explicitly when done. The wrapper preserves the established attachment across follow-ups instead of resending local-launch defaults, and live-checks the URL before later page reads or interactions because an attached browser can change tabs outside Pi.
|
|
603
|
-
- Do not treat an arbitrary `--session` name alone as persisted auth after `close`, `quit`, or `exit`. Wrapper-owned implicit sessions automatically use a Pi-transcript- and Git-checkout-generation-scoped `AGENT_BROWSER_RESTORE` key so cookies and web storage can survive relaunch, reload, and `/resume`; disable that convenience with `PI_AGENT_BROWSER_MANAGED_SESSION_RESTORE=0`. Caller-selected sessions, restore/state paths, profiles, upstream config, file access, launch arguments, environment variables, and close arguments pass through unchanged. `session list` and `state list` keep all upstream rows and restore identifiers visible. The wrapper does not reserve `piab-*` names or reject cross-checkout/local paths. Automatic restore still validates its own checkout/storage identity and coordinates same-daemon reuse so it cannot mix the wrapper's restore pools.
|
|
604
|
-
- Caller-owned explicit sessions are live-checked with `get url` before content-bearing reads or interactions. Missing or stale transcript page state is not treated as proof of a target; if the live URL cannot be verified, the requested content command does not run. Calls to the same actual socket context and effective canonical namespace/session are serialized across cooperating updated Pi processes; explicit namespace argv overrides `AGENT_BROWSER_NAMESPACE`, including an explicit empty default from that probe through any semantic-action snapshot and the requested command, while different caller-owned sessions remain independent. Raw non-bail batches are rejected when a failed navigation could leave an unverified target before later content; use exact `batch --bail` or split navigation from content. Nested `batch` steps are rejected, and raw batch command strings mirror upstream's ASCII-space tokenizer, including its single/double-quote and backslash handling, without splitting on other Unicode whitespace.
|
|
605
|
-
- Prefer page actions and storage checks over cookie dumps. `cookies get` can expose real profile cookies.
|
|
606
|
-
- Prefer `auth save --password-stdin` over putting passwords in `args`; the wrapper only accepts caller `stdin` for `batch`, `eval --stdin`, and `auth save --password-stdin` (advanced QA and source tools generate their own batch stdin).
|
|
607
|
-
- Use `state save <path>` / `state load <path>` for portable test state. `state save` is reported as a file artifact with verification metadata; if an upstream-successful artifact command reports a non-pending file path that the wrapper cannot find or did not update during this command, the tool fails with `failureCategory: "artifact-missing"` instead of treating missing/stale evidence as durable. `state load` may mention a path but is not treated as a newly saved artifact.
|
|
608
|
-
- `cookies set --curl <file>` treats its operand as a file path, not a cookie value: invocation echoes and failed batch diagnostics keep ordinary import paths visible. Inline cookie values and secret-bearing flags or path text still redact.
|
|
609
|
-
- Treat `cookies get`, `storage local|session`, `state show`, and `auth show` output as sensitive. `state show` is presented as saved-state metadata only, and cookie/localStorage/sessionStorage values are redacted from structured details. The native presentation summarizes and redacts credential-like values while allowing benign primitive storage values to aid local QA, but avoid requesting broad dumps unless the task needs them.
|
|
610
|
-
- Use `dialog status`, `dialog accept [text]`, `dialog dismiss`, and `frame <selector|main>` through native `args`; dialog commands use a shorter wrapper timeout and timed-out interactions add `inspect-dialog-after-timeout` / `dismiss-dialog-after-timeout` / fresh-session recovery actions so a blocking alert/prompt does not burn the full default watchdog. Use exact `confirm <id>` / `deny <id>` next actions for guarded-action confirmations.
|
|
611
|
-
|
|
612
|
-
Safe stateful examples:
|
|
613
|
-
|
|
614
|
-
```json
|
|
615
|
-
{ "args": ["auth", "save", "demo", "--password-stdin"], "stdin": "password from the user-approved secret source" }
|
|
616
|
-
{ "args": ["auth", "login", "demo"] }
|
|
617
|
-
{ "args": ["state", "save", "/tmp/demo-state.json"] }
|
|
618
|
-
{ "args": ["state", "load", "/tmp/demo-state.json"], "sessionMode": "fresh" }
|
|
619
|
-
{ "args": ["cookies", "set", "theme", "dark", "--url", "https://example.com"] }
|
|
620
|
-
{ "args": ["storage", "local", "get", "theme"] }
|
|
621
|
-
{ "args": ["dialog", "accept", "prompt text"] }
|
|
622
|
-
{ "args": ["frame", "main"] }
|
|
623
|
-
```
|
|
624
|
-
|
|
625
|
-
Example explicit session plus profile launch:
|
|
626
|
-
|
|
627
|
-
```json
|
|
628
|
-
{
|
|
629
|
-
"args": ["--session", "auth-flow", "--profile", "Default", "open", "https://example.com/account"]
|
|
630
|
-
}
|
|
631
|
-
```
|
|
632
|
-
|
|
633
|
-
## React, SPA, and first-navigation setup
|
|
634
|
-
|
|
635
|
-
React and SPA tooling from upstream `agent-browser` is passed through directly.
|
|
636
|
-
|
|
637
|
-
Launch React introspection before first navigation:
|
|
638
|
-
|
|
639
|
-
```json
|
|
640
|
-
{ "args": ["open", "--enable", "react-devtools", "https://example.com"], "sessionMode": "fresh" }
|
|
641
|
-
{ "args": ["react", "tree"] }
|
|
642
|
-
{ "args": ["react", "inspect", "<fiberId>"] }
|
|
643
|
-
{ "args": ["react", "renders", "start"] }
|
|
644
|
-
{ "args": ["react", "renders", "stop"] }
|
|
645
|
-
{ "args": ["react", "suspense", "--only-dynamic"] }
|
|
646
|
-
```
|
|
647
|
-
|
|
648
|
-
Use SPA and Web Vitals helpers as normal command tokens:
|
|
649
|
-
|
|
650
|
-
```json
|
|
651
|
-
{ "args": ["pushstate", "/dashboard"] }
|
|
652
|
-
{ "args": ["vitals", "https://example.com"] }
|
|
653
|
-
```
|
|
654
|
-
|
|
655
|
-
For setup that must happen before first navigation, open a blank fresh page, stage routes/cookies/scripts, then navigate:
|
|
656
|
-
|
|
657
|
-
```json
|
|
658
|
-
{ "args": ["open"], "sessionMode": "fresh" }
|
|
659
|
-
{ "args": ["network", "route", "**/*.js", "--abort", "--resource-type", "script"] }
|
|
660
|
-
{ "args": ["cookies", "set", "--curl", "/path/to/cookies.txt", "--domain", "example.com"] }
|
|
661
|
-
{ "args": ["navigate", "https://example.com"] }
|
|
662
|
-
```
|
|
663
|
-
|
|
664
|
-
## Proof and verification
|
|
665
|
-
|
|
666
|
-
`npm run docs` checks that generated playbook fragments and command-reference baseline blocks match their canonical sources (`extensions/agent-browser/lib/playbook.ts` and `scripts/agent-browser-capability-baseline.mjs`) without invoking upstream `agent-browser`.
|
|
667
|
-
|
|
668
|
-
The local verification gate is:
|
|
669
|
-
|
|
670
|
-
```bash
|
|
671
|
-
npm run verify
|
|
672
|
-
```
|
|
673
|
-
|
|
674
|
-
For a fast TypeScript-only iteration loop (same `tsc --noEmit` as the default gate, without docs drift checks, unit tests, or live upstream command-reference sampling):
|
|
675
|
-
|
|
676
|
-
```bash
|
|
677
|
-
npm run typecheck
|
|
678
|
-
```
|
|
679
|
-
|
|
680
|
-
The full `npm run verify` gate runs:
|
|
681
|
-
|
|
682
|
-
- generated playbook/documentation drift checks
|
|
683
|
-
- a clean build of generated `dist/` runtime files
|
|
684
|
-
- `tsc --noEmit`
|
|
685
|
-
- the test suite
|
|
686
|
-
- command-reference baseline checks
|
|
687
|
-
- live command-reference verification against the targeted installed upstream `agent-browser`
|
|
688
|
-
|
|
689
|
-
Step order and which subprocesses run live in [`scripts/project.mjs`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/scripts/project.mjs); [`test/project-verify.test.ts`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/test/project-verify.test.ts) locks default, `pre-pr`, `release`, `startup-profile`, `real-upstream`, `dogfood`, `platform-target`, `platform-smoke`, `package-pi`, and combined-docs orchestration so a gate cannot disappear accidentally. Run `npm run verify -- --help` for opt-in modes and supported passthrough flags.
|
|
690
|
-
|
|
691
|
-
For larger local handoffs or PR-ready confidence before expensive release/lifecycle/platform gates, run:
|
|
692
|
-
|
|
693
|
-
```bash
|
|
694
|
-
npm run verify -- pre-pr
|
|
695
|
-
```
|
|
696
|
-
|
|
697
|
-
That mode composes the full default gate with `npm run verify -- package`, so package contents and forbidden repo-only files are checked without launching Pi lifecycle, Crabbox, or live dogfood flows. Package modes build through npm `prepare`; lifecycle and startup-profile build in their focused scripts; default and platform-target build before consuming `dist/`, so clean checkouts do not validate stale or missing compiled output. The same `prepare` script owns GitHub/source installs; when Pi installs with `npm install --omit=dev`, it installs the source-build dev dependencies with lifecycle scripts disabled before building the ignored `dist/` entrypoint that Pi loads.
|
|
698
|
-
|
|
699
|
-
The opt-in startup profiler measures only the package extension entrypoint import plus factory registration in fresh Node processes. It intentionally does **not** launch Pi, tmux, mise, npm, browsers, or `agent-browser`; full Pi TUI ready-prompt profiling proved too invasive for routine verification on the operator machine. Run it after package entrypoint, generated runtime, or top-level import changes:
|
|
700
|
-
|
|
701
|
-
```bash
|
|
702
|
-
npm run build
|
|
703
|
-
npm run verify -- startup-profile --samples 3
|
|
704
|
-
```
|
|
705
|
-
|
|
706
|
-
Reports are written to `.artifacts/startup-profile/latest.json` even when the budget fails; `--json` still prints the complete report, and failure retains a nonzero exit. Each sample records its fresh child PID, aggregate and main-thread CPU time, child-lifetime resource counters, hardware allocation, post-sample load, and compiled-entrypoint/measurement-owner/selected SDK and CLI fingerprints when available. CPU baselines enclose the unchanged import/factory clock; resource, hardware, load and fingerprint collection happens afterward. Aggregate CPU includes worker threads and can exceed wall time, so do not subtract it from wall time or treat these observations as proof of contention. The existing regression test emits all three receipts before enforcing the unchanged maximum below 250 ms. Reports retain the safety block confirming no Pi, tmux, mise, npm, browser, or `agent-browser` subprocesses were launched.
|
|
707
|
-
|
|
708
|
-
The opt-in real-upstream suite is separate because it drives a real browser installation:
|
|
55
|
+
For a trial without a saved package setting, run:
|
|
709
56
|
|
|
710
57
|
```bash
|
|
711
|
-
|
|
712
|
-
```
|
|
713
|
-
|
|
714
|
-
That mode sets `PI_AGENT_BROWSER_REAL_UPSTREAM=1` and runs `test/agent-browser.real-upstream-contract.test.ts` against the real `agent-browser` on `PATH` (the stable version must meet the 0.35.0 floor; current command-reference validation targets the recommended 0.38.1 capability baseline). It covers inspection, skills, experimental WebMCP list/invoke/result/cancel plus `--no-webmcp`, and a broad core interaction and navigation matrix on localhost fixtures (including off-viewport click, frame-scoped selector/wait/click behavior, form command fixes, `batch` stdin, and `pushstate`), plus `vitals`, network route/requests/HAR, diff snapshot/screenshot/url, trace/profiler, console/errors/highlight, stream enable/status/disable, `cookies set --curl`, a `react tree` missing-renderer path, and `wait --download` with the on-disk caveat documented in release notes. The harness uses a throwaway temp `HOME` and dedicated socket/screenshot directories so the run does not touch your normal browser profile paths. Browser-opening or credential-dependent families such as `inspect`, `dashboard`, `chat`, provider clouds, and OS clipboard flows stay in fake-upstream or manual validation unless a safe deterministic fixture is added. For prerequisites, isolation details, and troubleshooting, see [`docs/RELEASE.md`](docs/RELEASE.md#real-upstream-contract-validation).
|
|
715
|
-
|
|
716
|
-
A deterministic host-only live-browser wrapper smoke is available without an LLM choosing tool calls:
|
|
717
|
-
|
|
718
|
-
```bash
|
|
719
|
-
npm run verify -- dogfood
|
|
720
|
-
```
|
|
721
|
-
|
|
722
|
-
That mode clean-builds the package, then drives persistent browser code, advanced QA/actions, native batch, screenshot artifact verification, and session close against a deterministic local fixture. It complements, but does not replace, the interactive Pi/tmux release dogfood in [`docs/RELEASE.md`](docs/RELEASE.md#pre-release-checks).
|
|
723
|
-
|
|
724
|
-
Cross-platform release coverage uses Crabbox to run macOS, Ubuntu Linux, and native Windows target suites; see [`docs/platform-smoke.md`](docs/platform-smoke.md) for the required matrix, standalone coverage (`npm run smoke:platform:all` and per-target `smoke:platform:macos` / `:ubuntu` / `:windows-native`), and artifact/lease inspection. The release gate is:
|
|
725
|
-
|
|
726
|
-
```bash
|
|
727
|
-
npm run doctor
|
|
728
|
-
npm run check:platform-smoke
|
|
729
|
-
npm run smoke:platform:ubuntu-image
|
|
730
|
-
npm run smoke:platform:doctor
|
|
731
|
-
npm run verify -- release
|
|
58
|
+
pi --no-extensions -e npm:pi-agent-browser-native
|
|
732
59
|
```
|
|
733
60
|
|
|
734
|
-
|
|
61
|
+
This command loads this extension explicitly and disables automatic extension loading. Other Pi settings and resource types still apply.
|
|
735
62
|
|
|
736
|
-
##
|
|
63
|
+
## Profiles and privacy
|
|
737
64
|
|
|
738
|
-
|
|
65
|
+
Ordinary calls use one named browser per root Pi session. A parent and its subagents share that browser. Separate roots get separate browser names.
|
|
739
66
|
|
|
740
|
-
|
|
741
|
-
2. The extension registers compact direct/code tools, an advanced-tool loader with five native default-inactive specialized tools (unless selected by the host), and the optional credential-backed `agent_browser_web_search` companion.
|
|
742
|
-
3. Tool calls are translated into upstream `agent-browser` CLI invocations with controlled args, stdin, environment, timeout, and session planning.
|
|
743
|
-
4. Upstream JSON/plain-text output is parsed into model-friendly content and structured details.
|
|
744
|
-
5. Screenshots, downloads, recordings, traces, profiles, and spill files are normalized as Pi-visible artifacts where possible.
|
|
745
|
-
6. Generated playbook text in docs and browser instructions stays aligned with `extensions/agent-browser/lib/playbook.ts`.
|
|
67
|
+
Shared root browsers survive Pi exit. Ask Pi to close your group's browser when finished.
|
|
746
68
|
|
|
747
|
-
|
|
69
|
+
Profile access exposes private page content to the model. That content can persist in transcripts and saved files. Use temporary profiles for tests.
|
|
748
70
|
|
|
749
|
-
|
|
71
|
+
Credential-like values are redacted, but ordinary page text stays visible. Verify the signed-in page after you select a profile or restore browser state.
|
|
750
72
|
|
|
751
|
-
|
|
752
|
-
- Recommends upstream `agent-browser` 0.38.1 and accepts stable runtimes at or above 0.35.0.
|
|
753
|
-
- Does not bundle `agent-browser`; users install it separately.
|
|
754
|
-
- Does not provide a human browser UI inside Pi; the primary UX is agent-invoked tool calls. `--headed` asks upstream to show a browser window, but the wrapper cannot yet prove that the window is visible on the user's desktop.
|
|
755
|
-
- Localhost means the browser host's loopback, not necessarily the shell/Pi host. If `http://localhost:<port>` or `http://127.0.0.1:<port>` fails with errors such as `ERR_EMPTY_RESPONSE`, use an environment-specific host-reachable HTTP(S) address. A `file://` fixture is supported when upstream browser launch settings allow it; use HTTP(S) only when the browser environment cannot reach the local file.
|
|
756
|
-
- A successful upstream `click` is not proof that the app handled the event. For state-changing flows, verify with a fresh snapshot, text/URL assertion, screenshot, or `pageChangeSummary` before reporting success.
|
|
757
|
-
- Real authenticated profile use is powerful but sensitive. Treat profile and cookie access as user-approved, task-specific behavior.
|
|
758
|
-
- Wrapper tab/session recovery is best effort around observed upstream behavior, not a replacement for explicit profile/session design.
|
|
73
|
+
Ask for a headed browser if you need to watch a task or complete a login. Remote environments can keep the window off your desktop.
|
|
759
74
|
|
|
760
|
-
|
|
75
|
+
See [profiles and sessions](docs/REFERENCE.md#authenticatedprofile-workflows) for profile selection, restore limits, and cleanup.
|
|
761
76
|
|
|
762
|
-
|
|
77
|
+
## Optional search and settings
|
|
763
78
|
|
|
764
|
-
|
|
79
|
+
To enable web search, set `EXA_API_KEY` or `BRAVE_API_KEY` before you start Pi. Exa is the default when both keys work. Provider charges apply.
|
|
765
80
|
|
|
766
|
-
|
|
81
|
+
Set provider preferences and browser defaults in `~/.pi/config/pi-agent-browser-native/config.json`. Trusted projects can use `.pi/config/pi-agent-browser-native/config.json`. `PI_AGENT_BROWSER_CONFIG` selects an override file.
|
|
767
82
|
|
|
768
|
-
|
|
83
|
+
Malformed config files and search-provider JSON report errors without parser previews that could expose credentials, and search-provider HTTP failures report only the status with a fixed hint.
|
|
769
84
|
|
|
770
|
-
|
|
85
|
+
Read [configuration examples](docs/REFERENCE.md#optional-package-config-and-web-search) for secret references, provider selection, and profile defaults.
|
|
771
86
|
|
|
772
|
-
|
|
773
|
-
npm install
|
|
774
|
-
```
|
|
87
|
+
## Troubleshooting and limits
|
|
775
88
|
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
Checkout-only extension smoke test:
|
|
89
|
+
Run the read-only health check:
|
|
779
90
|
|
|
780
91
|
```bash
|
|
781
|
-
|
|
92
|
+
npm exec --package pi-agent-browser-native -- pi-agent-browser-doctor
|
|
782
93
|
```
|
|
783
94
|
|
|
784
|
-
|
|
95
|
+
It checks the browser CLI, supported versions, and duplicate Pi package sources. It does not change settings.
|
|
785
96
|
|
|
786
|
-
|
|
97
|
+
This is a pre-1.0 package. Windows has unresolved browser session-locking and restored Electron cleanup test failures. Release tests do not cover Android. Electron desktop tools do not apply to Android apps. See the [support matrix](docs/SUPPORT_MATRIX.md) for details.
|
|
787
98
|
|
|
788
|
-
|
|
99
|
+
For recordings, install `ffmpeg` with the encoder for your WebM or MP4 output. Verify the completed file after you stop the recording.
|
|
789
100
|
|
|
790
|
-
|
|
791
|
-
npm run verify -- lifecycle
|
|
792
|
-
```
|
|
101
|
+
For remote browsers, use a URL that the browser host can reach. `localhost` refers to the browser host.
|
|
793
102
|
|
|
794
|
-
|
|
103
|
+
For stale references or unexpected tabs, follow the returned recovery actions. Take a fresh snapshot before another interaction. A snapshot inside a batch does not protect refs used after a later page-changing step; add another snapshot after that step or split the batch.
|
|
795
104
|
|
|
796
|
-
|
|
105
|
+
## More documentation
|
|
797
106
|
|
|
798
|
-
|
|
107
|
+
- [Detailed reference](docs/REFERENCE.md): platform setup, browser examples, compatibility notes, and development checks.
|
|
108
|
+
- [Tool contract](docs/TOOL_CONTRACT.md): exact inputs, results, code limits, and image geometry.
|
|
109
|
+
- [Electron guide](docs/ELECTRON.md): desktop discovery, isolated launch, and cleanup.
|
|
110
|
+
- [Session storage](docs/SESSION_STORAGE.md): replay, large histories, and session conversion.
|
|
111
|
+
- [Architecture](docs/ARCHITECTURE.md): extension design.
|
|
112
|
+
- [Code quality](docs/CODE_QUALITY.md) and [release process](docs/RELEASE.md): maintainer procedures.
|
|
799
113
|
|
|
800
|
-
|
|
801
|
-
npm run verify -- package-pi
|
|
802
|
-
pi --no-extensions -e npm:pi-agent-browser-native@<version>
|
|
803
|
-
```
|
|
114
|
+
## License
|
|
804
115
|
|
|
805
|
-
|
|
806
|
-
|
|
807
|
-
These sections are generated from `extensions/agent-browser/lib/playbook.ts`. Run `npm run docs -- playbook write` after changing the canonical playbook source.
|
|
808
|
-
|
|
809
|
-
<!-- agent-browser-playbook:start inspection -->
|
|
810
|
-
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
811
|
-
|
|
812
|
-
Native inspection calls use the `agent_browser` tool shape, not shell-like direct-binary commands:
|
|
813
|
-
|
|
814
|
-
- { "args": ["--help"] }
|
|
815
|
-
- { "args": ["--version"] }
|
|
816
|
-
|
|
817
|
-
These calls return plain text and stay stateless: the extension does not inject its implicit session and does not let inspection consume the managed-session slot needed for later profile, session, CDP, state, auto-connect, or provider-backed launches.
|
|
818
|
-
<!-- agent-browser-playbook:end inspection -->
|
|
819
|
-
|
|
820
|
-
<!-- agent-browser-playbook:start wrapper-tab-recovery -->
|
|
821
|
-
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
822
|
-
|
|
823
|
-
- After open/goto/navigate calls with --profile, --restore, --session-name, or --state, agent_browser best-effort re-selects the tab whose URL matches the returned page when restored tabs steal focus during launch or reconnect.
|
|
824
|
-
- After confirmed shutdown of an automatically restored managed session, the wrapper retains its complete recorded URL, including the fragment, until the first current-page operation (including get url and reload). Non-page calls such as tab list may start a daemon without fulfilling that reopen; explicit URL reads leave the managed browser and pending reopen untouched. The wrapper uses native open once, verifies the observed tab, and discards old refs/frame scope; it does not restore unsaved forms, JavaScript memory, or history. Explicit navigation, caller-owned/attached sessions, and restore-disabled sessions are not auto-reopened.
|
|
825
|
-
- For a still-live browser after tab drift or resume, the wrapper verifies/selects the intended tab before ref/semantic helpers and page commands; failed selection stops the call without navigating. Local commands, read <url>, URL a11y/vitals, diff url, window new, and explicit tab/navigation/connection/state recovery do not require the prior tab. Batch checks follow effective rows past non-page prefixes and stop at explicit context changes, preserving caller argv/stdin and continue-on-error behavior. Same-tab reselection is avoided because it clears refs. Use exact batch --bail for fail-fast, not --bail=<value>. Routine same-session calls skip tab-list preflights.
|
|
826
|
-
- For sessions with observed tab-drift risk, after a successful command on a known target tab, agent_browser also best-effort restores that intended tab if a restored/background tab steals focus after the command completes. Routine same-session commands skip focus correction; observed pages without a native targetId use tab list to establish their identity.
|
|
827
|
-
- If a known session target unexpectedly reports about:blank, agent_browser best-effort re-selects the prior intended target when it still exists; if recovery fails, it records the observed about:blank target and reports exact recovery guidance instead of treating the prior page as active.
|
|
828
|
-
- If upstream reports tab_gone, the pinned bound tab is gone; use details.nextActions (tab list / tab new) instead of assuming another tab is yours.
|
|
829
|
-
|
|
830
|
-
<!-- agent-browser-playbook:end wrapper-tab-recovery -->
|
|
831
|
-
|
|
832
|
-
## Project map
|
|
833
|
-
|
|
834
|
-
| Path | Purpose |
|
|
835
|
-
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
|
836
|
-
| `extensions/agent-browser/index.ts` | Native Pi factory wiring; packed installs load compiled `dist/extensions/agent-browser/index.js` |
|
|
837
|
-
| `extensions/agent-browser/lib/runtime.ts` | Stable public exports for pure planning, session identity/replay, timeout, and redaction helpers |
|
|
838
|
-
| `extensions/agent-browser/lib/orchestration/` | Direct/code admission, lifecycle, browser preparation/output, and Electron host ownership |
|
|
839
|
-
| `extensions/agent-browser/lib/results/` | Model-facing result rendering and error guidance |
|
|
840
|
-
| `extensions/agent-browser/lib/playbook.ts` | Canonical generated agent/browser guidance |
|
|
841
|
-
| `scripts/agent-browser-target.mjs` | Canonical recommended target and minimum supported stable version shared by runtime and build-time checks |
|
|
842
|
-
| `scripts/agent-browser-capability-baseline.mjs` | Help samples and doc/token inventory for drift checks; imports the canonical target version |
|
|
843
|
-
| `scripts/check-command-reference-baseline.mjs` | Regenerates or verifies HTML-bounded baseline blocks in `docs/COMMAND_REFERENCE.md` (via `npm run docs -- command-reference …`) |
|
|
844
|
-
| `docs/COMMAND_REFERENCE.md` | Repo-readable native command reference |
|
|
845
|
-
| `docs/TOOL_CONTRACT.md` | Tool parameters, result shape, and behavior contract |
|
|
846
|
-
| `docs/ELECTRON.md` | Dedicated public guide for Electron desktop-app support |
|
|
847
|
-
| `docs/ARCHITECTURE.md` | Design decisions and implementation structure |
|
|
848
|
-
| `docs/CODE_QUALITY.md` | Strict lint/formatting policy, checker corrections, scope, and semantic exceptions |
|
|
849
|
-
| `docs/REQUIREMENTS.md` | Product requirements and constraints |
|
|
850
|
-
| `docs/RELEASE.md` | Release, package, and lifecycle verification workflow |
|
|
851
|
-
| `docs/platform-smoke.md` | Crabbox macOS, Ubuntu, and native Windows release gate |
|
|
852
|
-
| `docs/SUPPORT_MATRIX.md` | Current upstream support audit and release-readiness matrix |
|
|
853
|
-
| `test/` | Wrapper, runtime, presentation, lifecycle, and package tests |
|
|
854
|
-
|
|
855
|
-
## More docs
|
|
856
|
-
|
|
857
|
-
- [`AGENTS.md`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/AGENTS.md) — maintainer and agent runbooks, including upstream capability baseline rebaselining and Pi smoke testing in `tmux`
|
|
858
|
-
- [`docs/COMMAND_REFERENCE.md`](docs/COMMAND_REFERENCE.md) — full native command reference and upstream capability baseline
|
|
859
|
-
- [`docs/TOOL_CONTRACT.md`](docs/TOOL_CONTRACT.md) — exact tool contract
|
|
860
|
-
- [`docs/ELECTRON.md`](docs/ELECTRON.md) — Electron desktop-app guide
|
|
861
|
-
- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — how the wrapper is designed
|
|
862
|
-
- [`docs/CODE_QUALITY.md`](docs/CODE_QUALITY.md) — strict quality commands, coverage, and verified exceptions
|
|
863
|
-
- [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md) — product constraints and non-goals
|
|
864
|
-
- [`docs/RELEASE.md`](docs/RELEASE.md) — maintainer release workflow
|
|
865
|
-
- [`docs/SUPPORT_MATRIX.md`](docs/SUPPORT_MATRIX.md) — current upstream support matrix and closure evidence
|
|
866
|
-
|
|
867
|
-
## Where to go next
|
|
868
|
-
|
|
869
|
-
If you are a user, install the package and ask Pi to open a public page with `agent_browser`.
|
|
870
|
-
|
|
871
|
-
If you are evaluating the implementation, read [`extensions/agent-browser/index.ts`](https://github.com/fitchmultz/pi-agent-browser-native/blob/main/extensions/agent-browser/index.ts), then run `npm run verify`.
|
|
116
|
+
[MIT](LICENSE). Copyright Mitch Fultz.
|