browser-pilot-cli 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +259 -406
- package/dist/cli.js +1089 -2550
- package/dist/daemon.js +125 -172
- package/docs/architecture/browser-pilot-platform-spec.md +443 -1047
- package/docs/plans/browser-capability-evolution.md +6 -6
- package/docs/plans/profile-context-routing.md +1 -1
- package/docs/plans/universal-agent-integration.md +121 -348
- package/docs/releases/v0.5.0.md +93 -0
- package/package.json +2 -8
- package/docs/integration/stdio-bridge.md +0 -694
- package/docs/integration/stdio-conformance.md +0 -51
- package/scripts/run-stdio-conformance.mjs +0 -698
package/README.md
CHANGED
|
@@ -1,488 +1,355 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Browser Pilot
|
|
2
2
|
|
|
3
|
-
Give
|
|
3
|
+
Give an AI Agent control of the user's real Chrome browser, including existing
|
|
4
|
+
logins, cookies, profiles, and extensions. No browser extension is required.
|
|
4
5
|
|
|
5
6
|
```bash
|
|
6
7
|
npm install -g browser-pilot-cli@latest
|
|
7
8
|
bp --version
|
|
8
9
|
```
|
|
9
10
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
embedded products should pin the tested version for reproducible builds.
|
|
13
|
-
The native release targets Apple Silicon macOS, x64 Linux, and x64 Windows;
|
|
14
|
-
Intel Mac is not supported.
|
|
11
|
+
Native releases support Apple Silicon macOS, x64 Linux, and x64 Windows. Intel
|
|
12
|
+
Mac is not supported.
|
|
15
13
|
|
|
16
14
|
## Agent Setup
|
|
17
15
|
|
|
18
|
-
### 1. Enable Chrome remote debugging
|
|
16
|
+
### 1. Enable Chrome remote debugging
|
|
19
17
|
|
|
20
|
-
Open `chrome://inspect/#remote-debugging` in Chrome
|
|
21
|
-
debugging.
|
|
22
|
-
|
|
18
|
+
Open `chrome://inspect/#remote-debugging` in Chrome 144 or newer and enable
|
|
19
|
+
remote debugging. Browser Pilot does not use command-line debugging flags or a
|
|
20
|
+
Chrome extension.
|
|
23
21
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
Check every installed supported browser and its exact setup state with:
|
|
22
|
+
Inspect installed browsers without requesting authorization:
|
|
27
23
|
|
|
28
24
|
```bash
|
|
29
25
|
bp browsers
|
|
30
26
|
```
|
|
31
27
|
|
|
32
|
-
|
|
33
|
-
|
|
28
|
+
Connect when browser work is needed:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
bp connect
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Chrome may display one Allow dialog for the browser connection. Browser
|
|
35
|
+
discovery and background-service startup are passive. Concurrent clients share
|
|
36
|
+
one pending connection request, so they do not create an authorization loop.
|
|
34
37
|
|
|
35
|
-
|
|
36
|
-
embedders can pass the same selector to `browser-pilot bridge --stdio
|
|
37
|
-
--browser <selector>`; otherwise selection follows a stable platform order.
|
|
38
|
-
One connection covers every live Chrome Profile on that endpoint. When several
|
|
39
|
-
Profiles are open, `bp connect` lists them without creating a Pilot window. Use
|
|
40
|
-
the passive `bp profiles` inventory first. Run `bp profiles --identify` only
|
|
41
|
-
when account-aware names are needed; it briefly opens and closes one visible
|
|
42
|
-
`chrome://version` page per unidentified Profile. Profile and tab indexes are
|
|
43
|
-
one-based. Select with `bp profile <index>`, or pass `--profile` to
|
|
44
|
-
`bp open --new`.
|
|
38
|
+
### 2. Install the Agent skill
|
|
45
39
|
|
|
46
|
-
|
|
40
|
+
Claude Code:
|
|
47
41
|
|
|
48
|
-
|
|
49
|
-
```
|
|
42
|
+
```text
|
|
50
43
|
/plugin marketplace add https://github.com/relixiaobo/browser-pilot.git
|
|
51
44
|
/plugin install browser-pilot@browser-pilot-marketplace
|
|
52
45
|
```
|
|
53
46
|
|
|
54
|
-
|
|
47
|
+
Codex, Cursor, or VS Code Copilot:
|
|
48
|
+
|
|
55
49
|
```bash
|
|
56
50
|
npx skills add relixiaobo/browser-pilot --skill browser-pilot
|
|
57
51
|
```
|
|
58
52
|
|
|
59
|
-
|
|
53
|
+
OpenClaw or another Agent Skills-compatible host:
|
|
54
|
+
|
|
60
55
|
```bash
|
|
61
56
|
cp -r plugin/skills/browser-pilot ~/.agents/skills/
|
|
62
57
|
```
|
|
63
58
|
|
|
64
|
-
|
|
65
|
-
```bash
|
|
66
|
-
npx skills add relixiaobo/browser-pilot --skill browser-pilot
|
|
67
|
-
```
|
|
59
|
+
### 3. Ask for the browser task
|
|
68
60
|
|
|
69
|
-
|
|
61
|
+
- "Fill in the form I already opened."
|
|
62
|
+
- "Check my GitHub notifications."
|
|
63
|
+
- "Download the latest invoice and summarize it."
|
|
64
|
+
- "Open this report in a new tab and save it as PDF."
|
|
70
65
|
|
|
71
|
-
|
|
66
|
+
The Agent uses its existing shell tool to run `bp`. Browser Pilot does not add
|
|
67
|
+
native Agent tools, an MCP server, an SDK, or a persistent client protocol.
|
|
72
68
|
|
|
73
|
-
|
|
74
|
-
- "Go to Hacker News and summarize the top 5 posts"
|
|
75
|
-
- "Fill out the form on this page"
|
|
69
|
+
## Public Interface
|
|
76
70
|
|
|
77
|
-
|
|
71
|
+
Browser Pilot has one Agent integration surface:
|
|
78
72
|
|
|
79
|
-
|
|
73
|
+
```text
|
|
74
|
+
Agent -> Browser Pilot skill -> existing shell/command runner
|
|
75
|
+
-> bp CLI -> shared background service -> Chrome
|
|
76
|
+
```
|
|
80
77
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
- **Auto-snapshot** — Every action returns page state with numbered `[ref]` elements, so the agent always knows what's on screen
|
|
86
|
-
- **Adaptive page views** — Bounded read, text search, DOM metadata, page geometry, and annotated screenshots let an agent choose the smallest useful representation
|
|
87
|
-
- **Verified page primitives** — Scroll and dropdown tools return fresh state and typed evidence instead of requiring ad hoc JavaScript
|
|
88
|
-
- **Lightweight npm install** — About 200KB as an npm tarball. No bundled Chromium (unlike Playwright's 400MB+)
|
|
89
|
-
- **Rich editor support** — Works with contenteditable editors (Draft.js, ProseMirror, Quill, Slate) and Shadow DOM elements out of the box
|
|
78
|
+
The skill provides progressive disclosure and browser-specific operating
|
|
79
|
+
guidance. The CLI provides stable JSON, errors, state, files, waiting, and
|
|
80
|
+
recovery. The background service is private implementation detail shared by all
|
|
81
|
+
compatible CLI installations for the current OS user.
|
|
90
82
|
|
|
91
|
-
|
|
83
|
+
This same path supports both installation modes:
|
|
92
84
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
| **Login session reuse** | Yes | No | Depends | Yes |
|
|
97
|
-
| **Extension required** | No | No | No | No |
|
|
98
|
-
| **Element refs** | Numbered (accessibility tree) | Named refs (ARIA) | CSS selectors | Numbered (DOM) |
|
|
99
|
-
| **Auto-snapshot after action** | Yes | Yes | No | Yes |
|
|
100
|
-
| **Network interception** | Yes (block/mock/headers) | Yes | Yes | No |
|
|
101
|
-
| **Multi-browser** | Chromium-only | Chromium + Firefox + WebKit | Chromium-only | Chromium-only |
|
|
102
|
-
| **Dialog handling** | Explicit | Automatic | Manual | Automatic |
|
|
103
|
-
| **JSON output** | Default | MCP structured | MCP structured | Python objects |
|
|
104
|
-
| **File upload** | Auto-detect input | Yes | No | Yes |
|
|
85
|
+
- An Agent installs `browser-pilot-cli` itself.
|
|
86
|
+
- A product bundles a pinned CLI and places its directory on the Agent command
|
|
87
|
+
environment's `PATH`.
|
|
105
88
|
|
|
106
|
-
|
|
89
|
+
Tenon, OpenClaw, Codex, and other shell-capable Agents need no Browser
|
|
90
|
+
Pilot-specific runtime integration. See
|
|
91
|
+
[the embedding reference](plugin/skills/browser-pilot/references/embedding.md).
|
|
107
92
|
|
|
108
|
-
|
|
109
|
-
Agent (bash tool)
|
|
110
|
-
│ bp open / bp click / bp eval ...
|
|
111
|
-
▼
|
|
112
|
-
CLI Process ──── HTTP/Unix Socket ──── Daemon Process (persistent)
|
|
113
|
-
│
|
|
114
|
-
│ WebSocket (CDP)
|
|
115
|
-
▼
|
|
116
|
-
Chrome (your browser, your Profiles)
|
|
117
|
-
├── Eligible user tabs (selectable)
|
|
118
|
-
└── Managed windows (Broker-owned)
|
|
119
|
-
```
|
|
93
|
+
## Browser Scope
|
|
120
94
|
|
|
121
|
-
|
|
122
|
-
supervisor owns the Broker's single browser-level CDP WebSocket and proxies it
|
|
123
|
-
to the daemon. It also owns only Browser Pilot-created tabs so it can clean them
|
|
124
|
-
after a Broker crash without touching user-opened tabs. Controller identity is
|
|
125
|
-
returned to the embedding Host; Browser Pilot never injects a status border or
|
|
126
|
-
other control UI into the page DOM.
|
|
95
|
+
`bp tabs` exposes all eligible page tabs on the authorized browser endpoint:
|
|
127
96
|
|
|
128
|
-
|
|
97
|
+
- Browser Pilot-created managed tabs;
|
|
98
|
+
- eligible popups from managed tabs;
|
|
99
|
+
- eligible tabs the user opened in any live Chrome profile.
|
|
129
100
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
101
|
+
The Agent can therefore fill a form or inspect a page the user already opened.
|
|
102
|
+
Browser-internal and extension-owned pages are excluded. Existing Chrome tab
|
|
103
|
+
groups do not limit tab control, whether expanded or collapsed, but Browser
|
|
104
|
+
Pilot cannot inspect or manage tab groups without an extension.
|
|
133
105
|
|
|
134
|
-
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
- [Stdio black-box conformance suite](docs/integration/stdio-conformance.md)
|
|
139
|
-
- [Reference consumer adapters](https://github.com/relixiaobo/browser-pilot/tree/main/examples/adapters),
|
|
140
|
-
including Tenon and OpenClaw lifecycle mappings
|
|
141
|
-
|
|
142
|
-
The public integration direction remains CLI-only: one-shot commands for direct
|
|
143
|
-
Agent use and a persistent `bridge --stdio` mode for products that embed the
|
|
144
|
-
official executable. Browser Pilot will not require an extension, Native SDK,
|
|
145
|
-
or MCP server. The repository's consumer adapters are source examples built
|
|
146
|
-
only on the stdio contract; they are intentionally excluded from the npm
|
|
147
|
-
package and do not create a second public library API.
|
|
148
|
-
|
|
149
|
-
The dedicated Pilot window remains the default managed tab set for independent
|
|
150
|
-
Agent work. The Broker architecture also includes all eligible user tabs in the
|
|
151
|
-
Agent's inventory without a separate grant step, so an Agent can operate a page
|
|
152
|
-
the user already opened. Invoking or exposing the tool is the authorization
|
|
153
|
-
boundary; products may apply their own approval UX or remove operations when
|
|
154
|
-
launching the bridge. Bulk cleanup remains limited to managed tabs, and user
|
|
155
|
-
tabs remain open when a session ends.
|
|
156
|
-
|
|
157
|
-
The `browser-pilot bridge --stdio` transport, Broker lifecycle, browser tool
|
|
158
|
-
dispatch, event replay, protected Artifacts, protocol 1.1 transport limit
|
|
159
|
-
negotiation, protocol 1.2 Chrome Profile routing, and protocol 1.3 verified
|
|
160
|
-
Profile identity and unambiguous result vocabulary are implemented. Browser
|
|
161
|
-
disconnect and explicit reconnect handling, scoped
|
|
162
|
-
download Artifacts, Workspace resource isolation, and typed watchdog events for
|
|
163
|
-
stalled navigation, selected-frame detach, pending dialogs, and repeated
|
|
164
|
-
browser-observable no-progress actions are also implemented. Bounded page
|
|
165
|
-
search, DOM metadata lookup, scrolling, native/ARIA dropdown operations,
|
|
166
|
-
Observation page geometry, and viewport screenshot annotations adapt the most
|
|
167
|
-
useful browser-use inspection patterns to the same scoped Broker contract. An
|
|
168
|
-
internal managed-target janitor is the sole owner of the browser-level CDP
|
|
169
|
-
connection, proxies daemon traffic over private IPC, and closes only
|
|
170
|
-
Broker-created tabs and their managed popup descendants if the daemon exits or
|
|
171
|
-
crashes. It does not persist target IDs or close user tabs. Broker startup and
|
|
172
|
-
browser discovery are passive. Only `bp connect` or the versioned
|
|
173
|
-
`browser.connect` tool requests Chrome authorization; concurrent clients share
|
|
174
|
-
one in-flight request, and a failed or dropped connection is never retried by a
|
|
175
|
-
timer.
|
|
176
|
-
Compatible Agent products reuse one per-user Broker through protocol
|
|
177
|
-
negotiation. Protected shutdown cannot terminate live embedded clients or
|
|
178
|
-
another CLI Agent's active Lease, and an
|
|
179
|
-
incompatible product must explicitly select a separate `BROWSER_PILOT_HOME`.
|
|
180
|
-
Global npm installation, local npm/npx use, and product-bundled absolute-path
|
|
181
|
-
launches are covered by distribution black-box tests. Native self-contained
|
|
182
|
-
release artifacts use one executable for the public CLI and its private Broker
|
|
183
|
-
and janitor roles, with no runtime download or system-Node fallback.
|
|
184
|
-
|
|
185
|
-
## Installation and Embedding
|
|
186
|
-
|
|
187
|
-
Agent-managed installation remains supported:
|
|
106
|
+
Browser Pilot does not implement task-intent permission prompts. Invoking the
|
|
107
|
+
tool gives the Agent the full eligible browser capability surface. An Agent
|
|
108
|
+
host may apply its own policy or approval UX, but that is outside the Browser
|
|
109
|
+
Pilot contract.
|
|
188
110
|
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
bp connect
|
|
193
|
-
bp tabs
|
|
194
|
-
```
|
|
111
|
+
User-opened tabs remain open when Agent state is released. `bp close --all`
|
|
112
|
+
closes only Browser Pilot-created tabs. `bp close` explicitly closes the
|
|
113
|
+
selected tab, including a user tab.
|
|
195
114
|
|
|
196
|
-
|
|
197
|
-
|
|
115
|
+
## Chrome Profiles
|
|
116
|
+
|
|
117
|
+
One authorized browser connection can expose several live Chrome profiles.
|
|
198
118
|
|
|
199
119
|
```bash
|
|
200
|
-
|
|
201
|
-
|
|
120
|
+
bp profiles
|
|
121
|
+
bp profiles --identify
|
|
122
|
+
bp profile 2
|
|
123
|
+
bp open https://example.com --new --profile 2
|
|
202
124
|
```
|
|
203
125
|
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
`browser-pilot-plugin-<version>.tgz`, its checksum, and a versioned release index
|
|
219
|
-
that binds the npm version, protocol range, compatible skill/plugin range,
|
|
220
|
-
tested CLI version, and the three supported native assets. Consumers should
|
|
221
|
-
verify that index before bundling. No `darwin-x64` asset is produced.
|
|
222
|
-
|
|
223
|
-
All three forms use the same protocol and compatible per-user Broker. Use a
|
|
224
|
-
distinct absolute `BROWSER_PILOT_HOME` only when a product deliberately needs
|
|
225
|
-
an incompatible isolated Broker.
|
|
226
|
-
|
|
227
|
-
Products embedding `bridge --stdio` already receive isolated Connections,
|
|
228
|
-
Workspaces, and Leases. When several independent Agents instead invoke the
|
|
229
|
-
one-shot CLI directly, assign each integration a distinct stable namespace:
|
|
126
|
+
`bp profiles` is passive and includes representative tabs. Use
|
|
127
|
+
`--identify` only when account-aware identity is needed; it briefly opens and
|
|
128
|
+
closes a visible `chrome://version` page per unidentified profile and accepts
|
|
129
|
+
identity only after exact profile-path verification. Browser Pilot reports
|
|
130
|
+
verified profile/account names rather than forcing the Agent to speak in
|
|
131
|
+
ambiguous "Profile 1" labels.
|
|
132
|
+
|
|
133
|
+
Profile selection controls where new managed tabs are created. It does not
|
|
134
|
+
grant or restrict access to existing eligible tabs.
|
|
135
|
+
|
|
136
|
+
## Multi-Agent State
|
|
137
|
+
|
|
138
|
+
Every independent Agent must use one stable key across all short-lived CLI
|
|
139
|
+
processes:
|
|
230
140
|
|
|
231
141
|
```bash
|
|
232
|
-
BROWSER_PILOT_CLIENT_KEY=
|
|
233
|
-
bp
|
|
142
|
+
export BROWSER_PILOT_CLIENT_KEY=product.install.agent-01
|
|
143
|
+
bp tabs
|
|
144
|
+
bp snapshot
|
|
234
145
|
```
|
|
235
146
|
|
|
236
|
-
|
|
237
|
-
refs, auth,
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
## Commands
|
|
241
|
-
|
|
242
|
-
### Core Loop
|
|
243
|
-
|
|
244
|
-
| Command | Returns | Description |
|
|
245
|
-
|---------|---------|-------------|
|
|
246
|
-
| `bp open <url>` | snapshot | Navigate to URL |
|
|
247
|
-
| `bp open <url> --new --profile <selector>` | snapshot | Create managed work in one live Chrome Profile |
|
|
248
|
-
| `bp snapshot` | snapshot | Get interactive elements |
|
|
249
|
-
| `bp read [selector]` | text | Get bounded readable page/region content |
|
|
250
|
-
| `bp search <text>` | matches | Find bounded visible text and nearby context |
|
|
251
|
-
| `bp find <selector>` | elements | Inspect bounded DOM metadata and requested attributes |
|
|
252
|
-
| `bp scroll [direction]` | snapshot | Scroll page/element/text and return fresh state |
|
|
253
|
-
| `bp dropdown <target>` | options | List native or exposed ARIA dropdown options |
|
|
254
|
-
| `bp select <target> <option>` | snapshot | Select and verify a dropdown option |
|
|
255
|
-
| `bp click <ref>` | snapshot | Click element by ref number (`--double`, `--right`) |
|
|
256
|
-
| `bp click --xy x,y` | snapshot | Click at viewport coordinates (canvas, maps) |
|
|
257
|
-
| `bp locate <selector>` | coords | Get element center x,y + size (for `click --xy`) |
|
|
258
|
-
| `bp type <ref> <text>` | snapshot | Type into element (`--clear`, `--submit`) |
|
|
259
|
-
| `bp keyboard <text>` | snapshot | Type via keyboard events (`--click`, `--clear`) |
|
|
260
|
-
| `bp press <key>` | snapshot | Press key (Enter, Escape, Control+a, Meta+c) |
|
|
261
|
-
| `bp eval [js]` | value | Run JavaScript (escape hatch for anything) |
|
|
262
|
-
|
|
263
|
-
### Utilities
|
|
264
|
-
|
|
265
|
-
| Command | Description |
|
|
266
|
-
|---------|-------------|
|
|
267
|
-
| `bp screenshot [file]` | Capture screenshot (`--full`, `--selector`, `--annotate [refs]`) |
|
|
268
|
-
| `bp pdf [file]` | Save page as PDF (`--landscape`) |
|
|
269
|
-
| `bp cookies [domain]` | View cookies (includes HttpOnly) |
|
|
270
|
-
|
|
271
|
-
### Edge Cases
|
|
272
|
-
|
|
273
|
-
| Command | Description |
|
|
274
|
-
|---------|-------------|
|
|
275
|
-
| `bp upload <filepath>` | Upload file (auto-finds `<input type="file">`) |
|
|
276
|
-
| `bp auth <user> <pass>` | Set HTTP Basic Auth credentials (`--clear`) |
|
|
277
|
-
| `bp frame [index]` | List or switch iframe context (0 = top) |
|
|
278
|
-
| `bp dialogs` | List pending JavaScript dialogs |
|
|
279
|
-
| `bp dialog <id> --accept\|--dismiss` | Explicitly respond to a dialog (`--prompt`) |
|
|
280
|
-
|
|
281
|
-
Dialogs remain pending until explicitly accepted or dismissed. One-shot CLI
|
|
282
|
-
dialogs are scoped to the compatibility Workspace and isolated from embedded
|
|
283
|
-
Broker clients.
|
|
284
|
-
|
|
285
|
-
Run `bp tabs` to list Pilot-managed tabs, their popups, and eligible tabs the
|
|
286
|
-
user opened elsewhere across all live Profiles in the same browser endpoint.
|
|
287
|
-
Extension-owned and browser-internal pages are excluded. Each JSON entry
|
|
288
|
-
includes its opaque `profileContextId`; `bp tab <n>` selects any listed tab for
|
|
289
|
-
the current Agent Lease. Its `selected` field is not a claim about Chrome's
|
|
290
|
-
foreground tab or operating-system focus.
|
|
291
|
-
|
|
292
|
-
### Network
|
|
293
|
-
|
|
294
|
-
| Command | Description |
|
|
295
|
-
|---------|-------------|
|
|
296
|
-
| `bp net` | List recent requests (`--url`, `--method`, `--status`, `--type`) |
|
|
297
|
-
| `bp net show <id>` | Full request/response details (`--save <file>`) |
|
|
298
|
-
| `bp net block <pattern>` | Block requests matching URL pattern |
|
|
299
|
-
| `bp net mock <pattern>` | Mock responses (`--body`, `--file`) |
|
|
300
|
-
| `bp net headers <pattern> <header...>` | Add/override request headers |
|
|
301
|
-
| `bp net rules` | List active interception rules |
|
|
302
|
-
| `bp net remove [id]` | Remove rule(s) (`--all`) |
|
|
303
|
-
| `bp net clear` | Clear captured request log |
|
|
304
|
-
|
|
305
|
-
### Session
|
|
306
|
-
|
|
307
|
-
| Command | Description |
|
|
308
|
-
|---------|-------------|
|
|
309
|
-
| `bp connect` | Connect to Chrome; create a Pilot window immediately only when Profile routing is unambiguous |
|
|
310
|
-
| `bp disconnect` | Release this CLI namespace; stop the daemon only when no other client or active Lease is live |
|
|
311
|
-
| `bp profiles` | Passively list live Chrome Profile contexts and representative tabs |
|
|
312
|
-
| `bp profiles --identify [--refresh]` | Explicitly verify Profile/account identity using temporary visible Chrome pages |
|
|
313
|
-
| `bp profile <index\|id\|label\|name\|email>` | Select a Profile for subsequent managed tabs |
|
|
314
|
-
| `bp tabs` | List all controllable tabs in the current browser |
|
|
315
|
-
| `bp tab <n>` | Select any listed managed or user tab for the current Agent Lease |
|
|
316
|
-
| `bp close` | Close the current tab (`--all` closes Pilot-managed tabs only) |
|
|
317
|
-
|
|
318
|
-
## Refs
|
|
319
|
-
|
|
320
|
-
Action commands return a snapshot of interactive elements, each with a `[ref]` number:
|
|
147
|
+
Or pass `--client-key` before the command. The key preserves the Agent's
|
|
148
|
+
selected profile, tab, frame, refs, auth, network rules, downloads, and command
|
|
149
|
+
history. Different keys are isolated and cannot overwrite one another.
|
|
321
150
|
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
[4] combobox "" ← <select> dropdown
|
|
327
|
-
[5] spinbutton "Quantity" ← <input type="number">
|
|
328
|
-
[6] button "Submit"
|
|
329
|
-
[7] checkbox "Agree" checked
|
|
330
|
-
[8] slider "Volume" ← <input type="range">
|
|
331
|
-
```
|
|
151
|
+
Browser Pilot keeps this state in background-service memory, not global
|
|
152
|
+
`state.json` or `refs.json` files. `bp disconnect` releases only the current
|
|
153
|
+
key. It stops the service only when no other Agent is using it and the invoking
|
|
154
|
+
executable owns that service process.
|
|
332
155
|
|
|
333
|
-
|
|
156
|
+
Compatible global, local, and product-bundled CLIs reuse the same per-user
|
|
157
|
+
service through protocol negotiation. Executable identity is required only for
|
|
158
|
+
stopping the whole service. Use a distinct absolute `BROWSER_PILOT_HOME` only
|
|
159
|
+
for a deliberately incompatible isolated installation.
|
|
334
160
|
|
|
335
|
-
|
|
336
|
-
action. Elements inside Shadow DOM are included automatically. Snapshot JSON
|
|
337
|
-
may also include a `page` block with viewport/document size, scroll position,
|
|
338
|
-
remaining pixels, and scroll percentages.
|
|
161
|
+
## Core Workflow
|
|
339
162
|
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
163
|
+
```bash
|
|
164
|
+
bp tabs # find existing user or managed tabs
|
|
165
|
+
bp tab 2 # select by current one-based index
|
|
166
|
+
bp open https://example.com --new # create independent managed work
|
|
167
|
+
bp snapshot # get controls with numbered refs
|
|
168
|
+
bp read main --limit 10000 # get bounded readable content
|
|
169
|
+
bp search "invoice total" # find text with nearby context
|
|
170
|
+
bp click 3 # act on a fresh ref
|
|
171
|
+
bp type 5 "hello" --clear # replace input and verify readback
|
|
172
|
+
bp press Enter # press a key and return fresh state
|
|
173
|
+
```
|
|
346
174
|
|
|
347
|
-
|
|
175
|
+
Action commands return fresh page state and typed browser-visible evidence.
|
|
176
|
+
Refs belong to the current tab, frame, and page state. Refresh them after
|
|
177
|
+
navigation, document changes, tab/frame switches, reconnect, or `stale_ref`.
|
|
348
178
|
|
|
349
|
-
|
|
179
|
+
Use dedicated page primitives before JavaScript:
|
|
350
180
|
|
|
351
|
-
```
|
|
352
|
-
|
|
181
|
+
```bash
|
|
182
|
+
bp find ".result" --attributes href
|
|
183
|
+
bp scroll --to-text "Payment details"
|
|
184
|
+
bp dropdown 4
|
|
185
|
+
bp select 4 "United States"
|
|
186
|
+
bp screenshot page.png --annotate 1,3,8
|
|
353
187
|
```
|
|
354
188
|
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
{"ok":false,"error":"Ref [99] is stale.","code":"stale_ref","retryable":false,"context":{"field":"ref"},"hint":"Run 'bp snapshot' to refresh element refs."}
|
|
358
|
-
```
|
|
189
|
+
`bp eval` remains an escape hatch when no dedicated command represents the
|
|
190
|
+
operation.
|
|
359
191
|
|
|
360
|
-
|
|
361
|
-
it does not make the same uncertain mutation safe to replay.
|
|
192
|
+
## Async Work and Recovery
|
|
362
193
|
|
|
363
|
-
|
|
194
|
+
Wait for browser-visible state instead of using shell sleeps:
|
|
364
195
|
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
196
|
+
```bash
|
|
197
|
+
bp --timeout 30000 wait --url "*/complete"
|
|
198
|
+
bp --timeout 30000 wait --text "Payment received"
|
|
199
|
+
bp --timeout 30000 wait --selector ".result"
|
|
200
|
+
bp --timeout 30000 wait --dialog
|
|
201
|
+
bp --timeout 120000 wait --download
|
|
202
|
+
bp --timeout 30000 wait --popup
|
|
203
|
+
```
|
|
370
204
|
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
`--annotate 1,3,8` to limit it. Annotation does not inject elements into the
|
|
374
|
-
page, runs in an isolated JavaScript world, and cannot be combined with full-page
|
|
375
|
-
or selector capture.
|
|
205
|
+
Inspect service and command state after a timeout, interrupted shell call, or
|
|
206
|
+
lost result:
|
|
376
207
|
|
|
377
|
-
|
|
208
|
+
```bash
|
|
209
|
+
bp status
|
|
210
|
+
bp commands --status accepted,dispatched,unknown_outcome
|
|
211
|
+
bp command <command-id>
|
|
212
|
+
bp cancel <command-id>
|
|
213
|
+
```
|
|
378
214
|
|
|
379
|
-
|
|
215
|
+
Attach a stable request identity before the first attempt when the caller may
|
|
216
|
+
need to recover the same intended operation:
|
|
380
217
|
|
|
381
218
|
```bash
|
|
382
|
-
bp
|
|
383
|
-
bp eval "history.forward()" # go forward
|
|
384
|
-
bp eval "location.reload()" # reload
|
|
385
|
-
bp eval "document.querySelector('h1').textContent" # extract text
|
|
386
|
-
bp eval "document.querySelector('div').innerHTML" # extract HTML
|
|
387
|
-
bp eval "JSON.stringify(localStorage)" # read storage
|
|
388
|
-
echo 'complex js here' | bp eval # stdin for complex JS
|
|
219
|
+
bp --request-id task-42-submit click 7
|
|
389
220
|
```
|
|
390
221
|
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
222
|
+
Reuse that ID only for recovery of the same intended call. Use a new ID for a
|
|
223
|
+
new action. Browser Pilot deduplicates stable requests and attempts
|
|
224
|
+
cancellation on CLI `SIGINT`/`SIGTERM`, but a dispatched mutation can still
|
|
225
|
+
have an `unknown_outcome`. Inspect the current page before retrying.
|
|
226
|
+
|
|
227
|
+
Machine errors always include stable `code` and `retryable` fields. Branch on
|
|
228
|
+
`code`, not English text.
|
|
394
229
|
|
|
395
|
-
##
|
|
230
|
+
## Files, Screenshots, PDFs, and Downloads
|
|
396
231
|
|
|
397
|
-
|
|
232
|
+
Use explicit task-owned paths or set a default directory:
|
|
398
233
|
|
|
399
234
|
```bash
|
|
400
|
-
|
|
401
|
-
bp
|
|
402
|
-
bp
|
|
235
|
+
export BROWSER_PILOT_OUTPUT_DIR=/absolute/task/output
|
|
236
|
+
bp screenshot
|
|
237
|
+
bp screenshot full.png --full
|
|
238
|
+
bp pdf report.pdf --landscape
|
|
239
|
+
bp downloads
|
|
240
|
+
bp download 1 invoice.pdf
|
|
241
|
+
bp net show 12 --save response.json
|
|
403
242
|
```
|
|
404
243
|
|
|
405
|
-
|
|
244
|
+
File results return an absolute `file` path, MIME type, byte size, and image
|
|
245
|
+
dimensions when available. An Agent can open that path with its normal image or
|
|
246
|
+
file-reading capability. CLI output remains bounded JSON and never embeds large
|
|
247
|
+
binary payloads.
|
|
406
248
|
|
|
407
|
-
|
|
249
|
+
Browser Pilot copies uploads and completed downloads into protected temporary
|
|
250
|
+
storage while processing them. It never removes the source upload or Chrome's
|
|
251
|
+
original download.
|
|
408
252
|
|
|
409
|
-
|
|
410
|
-
bp type 3 "new content" --clear # replace content in a rich text editor
|
|
411
|
-
```
|
|
253
|
+
## Command Groups
|
|
412
254
|
|
|
413
|
-
|
|
255
|
+
| Area | Commands |
|
|
256
|
+
|------|----------|
|
|
257
|
+
| Setup and state | `browsers`, `connect`, `status`, `disconnect` |
|
|
258
|
+
| Profiles and tabs | `profiles`, `profile`, `tabs`, `tab`, `open`, `close` |
|
|
259
|
+
| Observation | `snapshot`, `read`, `search`, `find`, `locate` |
|
|
260
|
+
| Interaction | `click`, `type`, `keyboard`, `press`, `scroll`, `dropdown`, `select`, `eval` |
|
|
261
|
+
| Async and recovery | `wait`, `commands`, `command`, `cancel` |
|
|
262
|
+
| Page context | `frame`, `dialogs`, `dialog` |
|
|
263
|
+
| Files and media | `upload`, `downloads`, `download`, `screenshot`, `pdf` |
|
|
264
|
+
| Browser data | `cookies`, `auth`, `net` |
|
|
414
265
|
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
266
|
+
Run `bp --help` or `bp <command> --help` for exact options.
|
|
267
|
+
|
|
268
|
+
## How It Works
|
|
269
|
+
|
|
270
|
+
```text
|
|
271
|
+
short-lived bp CLI processes
|
|
272
|
+
|
|
|
273
|
+
| private local HTTP/Unix socket
|
|
274
|
+
v
|
|
275
|
+
per-user Browser Pilot service
|
|
276
|
+
|
|
|
277
|
+
| one supervised CDP WebSocket per browser endpoint
|
|
278
|
+
v
|
|
279
|
+
Chrome profiles and eligible tabs
|
|
420
280
|
```
|
|
421
281
|
|
|
422
|
-
|
|
282
|
+
The service serializes actions per physical tab while allowing independent
|
|
283
|
+
tabs to progress concurrently. Each Agent receives isolated logical state and
|
|
284
|
+
opaque tab/ref addresses. The same physical user tab can be controlled by only
|
|
285
|
+
one Agent at a time.
|
|
423
286
|
|
|
424
|
-
|
|
425
|
-
|
|
287
|
+
An internal connection supervisor owns Browser Pilot-created tabs so it can
|
|
288
|
+
clean them up after a service crash without touching user-opened tabs. Browser
|
|
289
|
+
disconnect invalidates old tab, frame, and ref identities before reconnect.
|
|
426
290
|
|
|
427
|
-
|
|
428
|
-
bp dropdown 4
|
|
429
|
-
bp select 4 "Option 2"
|
|
430
|
-
```
|
|
291
|
+
See:
|
|
431
292
|
|
|
432
|
-
|
|
293
|
+
- [Current architecture](docs/architecture/browser-pilot-platform-spec.md)
|
|
294
|
+
- [Universal Agent integration plan](docs/plans/universal-agent-integration.md)
|
|
295
|
+
- [Browser capability evolution](docs/plans/browser-capability-evolution.md)
|
|
296
|
+
- [Chrome profile routing](docs/plans/profile-context-routing.md)
|
|
433
297
|
|
|
434
|
-
|
|
298
|
+
## Comparison
|
|
435
299
|
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
300
|
+
| | Browser Pilot | browser-use | Playwright MCP | Chrome DevTools MCP |
|
|
301
|
+
|---|---|---|---|---|
|
|
302
|
+
| Agent interface | Skill + CLI | Python library/Agent runtime | MCP tools | MCP tools |
|
|
303
|
+
| User's signed-in Chrome | Yes | Yes | Usually isolated browser | Depends on launch mode |
|
|
304
|
+
| Browser extension | No | No | No | No |
|
|
305
|
+
| Eligible user tabs | Yes | Browser-session dependent | Browser-session dependent | Yes |
|
|
306
|
+
| Progressive Agent guidance | Skill references | Library/Agent prompts | Tool descriptions | Tool descriptions |
|
|
307
|
+
| Action verification | Typed evidence + fresh state | State/history checks | Fresh snapshots | Manual inspection |
|
|
308
|
+
| Async recovery | Wait + command status + request IDs | Agent/controller lifecycle | Host/MCP lifecycle | Host/MCP lifecycle |
|
|
309
|
+
| Local media results | Paths + metadata | Library objects/paths | MCP image/resource results | MCP results |
|
|
310
|
+
| Network interception | Observe, block, mock, headers | Limited by controller | Supported | Supported |
|
|
311
|
+
|
|
312
|
+
Browser Pilot adopts several strong browser-use ideas without adopting its
|
|
313
|
+
Python Agent runtime: adaptive DOM/accessibility observation, fresh state after
|
|
314
|
+
actions, input readback, obstruction checks, stale-state invalidation,
|
|
315
|
+
watchdogs, and bounded outputs. The Agent interface remains the CLI.
|
|
316
|
+
|
|
317
|
+
## Product Embedding
|
|
318
|
+
|
|
319
|
+
Bundle the native CLI or `browser-pilot-cli` plus a pinned Node runtime. Verify
|
|
320
|
+
the versioned release index and checksums. Put the bundled command directory on
|
|
321
|
+
the Agent's `PATH`, install the complete skill, and inject:
|
|
322
|
+
|
|
323
|
+
```text
|
|
324
|
+
BROWSER_PILOT_CLIENT_KEY=<stable logical Agent identity>
|
|
325
|
+
BROWSER_PILOT_OUTPUT_DIR=<absolute task-owned directory>
|
|
458
326
|
```
|
|
459
327
|
|
|
460
|
-
|
|
328
|
+
Do not map every CLI command into a native tool. The Agent already has a shell
|
|
329
|
+
tool, the skill controls context growth, and `bp --help` provides runtime
|
|
330
|
+
discovery. Agent-managed and product-bundled installations therefore exercise
|
|
331
|
+
the exact same implementation and guidance.
|
|
461
332
|
|
|
462
|
-
|
|
463
|
-
|
|
333
|
+
Each release ships a plugin archive, native archives, checksums, and a release
|
|
334
|
+
index binding the CLI version, compatible skill range, protocol range, and
|
|
335
|
+
supported assets. No `darwin-x64` artifact is produced.
|
|
336
|
+
|
|
337
|
+
## Testing
|
|
464
338
|
|
|
465
339
|
```bash
|
|
466
|
-
npm test
|
|
467
|
-
npm
|
|
468
|
-
npm run test:
|
|
469
|
-
npm run test:distribution
|
|
470
|
-
npm run test:canary
|
|
471
|
-
npm run test:canary:strict
|
|
472
|
-
npm run test:integration # compatibility alias for strict canaries
|
|
473
|
-
npm run test:all # release gates plus non-blocking canaries
|
|
340
|
+
npm run test:unit # TypeScript build and all Node tests
|
|
341
|
+
npm test # unit tests plus local Playwright gates
|
|
342
|
+
npm run test:capabilities # isolated-Chrome quantitative capability gate
|
|
343
|
+
npm run test:distribution # packed global/local/bundled CLI verification
|
|
344
|
+
npm run test:canary # non-blocking real-site drift report
|
|
345
|
+
npm run test:canary:strict
|
|
474
346
|
```
|
|
475
347
|
|
|
476
|
-
Automated browser
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
files. Tests fail closed if that isolation is absent; they never attach to the
|
|
480
|
-
user's Chrome by default. A maintainer can explicitly opt into a destructive
|
|
481
|
-
manual compatibility run against the user's browser by setting
|
|
482
|
-
`BROWSER_PILOT_TEST_USER_CHROME=1`.
|
|
348
|
+
Automated browser tests use isolated temporary Chrome profiles and Broker homes.
|
|
349
|
+
They never attach to the user's Chrome unless a maintainer explicitly opts into
|
|
350
|
+
a controlled manual test.
|
|
483
351
|
|
|
484
|
-
|
|
485
|
-
SEA-capable Node 22.17.0 runtime:
|
|
352
|
+
Build and verify a native artifact with an SEA-capable Node runtime:
|
|
486
353
|
|
|
487
354
|
```bash
|
|
488
355
|
BROWSER_PILOT_SEA_NODE=/absolute/path/to/node npm run build:standalone
|
|
@@ -490,35 +357,21 @@ npm run verify:standalone
|
|
|
490
357
|
npm run package:standalone
|
|
491
358
|
```
|
|
492
359
|
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
360
|
+
The root `package.json` version is the release source of truth. The npm version
|
|
361
|
+
lifecycle synchronizes the lockfile, plugin manifests, skill compatibility
|
|
362
|
+
range, and marketplace entry:
|
|
496
363
|
|
|
497
364
|
```bash
|
|
498
365
|
npm version "$BROWSER_PILOT_RELEASE_VERSION" --no-git-tag-version
|
|
499
366
|
npm run release:check-version
|
|
500
|
-
node scripts/verify-release-version.mjs
|
|
501
367
|
```
|
|
502
368
|
|
|
503
|
-
The sync step updates the lockfile root, Agent plugin version and CLI peer
|
|
504
|
-
range, Claude plugin/marketplace manifests, and skill `compatibility.json`.
|
|
505
|
-
The current release becomes the skill's tested/minimum CLI version while the
|
|
506
|
-
upper bound remains the next major version. Browser Pilot keeps public one-shot
|
|
507
|
-
commands backward compatible within a major line, including additive `0.x`
|
|
508
|
-
releases before 1.0. Historical release notes and version-specific fixtures are
|
|
509
|
-
never rewritten.
|
|
510
|
-
|
|
511
|
-
The canary report is written to
|
|
512
|
-
`test-results/real-site-canary/report.json`. It distinguishes semantic drift,
|
|
513
|
-
third-party unavailability, and runner errors. Set
|
|
514
|
-
`BROWSER_PILOT_CANARY_REPORT` to use another report path.
|
|
515
|
-
|
|
516
369
|
## Requirements
|
|
517
370
|
|
|
518
|
-
- Chrome 144
|
|
519
|
-
- Node.js
|
|
520
|
-
- Remote debugging enabled
|
|
521
|
-
-
|
|
371
|
+
- Chrome 144+, Edge, Brave, or another supported Chromium browser
|
|
372
|
+
- Node.js 18 or newer for the npm distribution
|
|
373
|
+
- Remote debugging enabled at `chrome://inspect/#remote-debugging`
|
|
374
|
+
- Apple Silicon macOS, x64 Linux, or x64 Windows for native releases
|
|
522
375
|
|
|
523
376
|
## License
|
|
524
377
|
|