browser-pilot-cli 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +270 -354
- package/dist/cli.js +1285 -2864
- package/dist/daemon.js +1142 -887
- package/docs/architecture/browser-pilot-platform-spec.md +440 -1013
- package/docs/plans/browser-capability-evolution.md +6 -6
- package/docs/plans/profile-context-routing.md +35 -10
- package/docs/plans/universal-agent-integration.md +121 -335
- package/docs/plans/v0.3.0-stabilization.md +10 -4
- package/docs/releases/v0.2.0.md +53 -0
- package/docs/releases/v0.2.1.md +13 -0
- package/docs/releases/v0.2.2.md +18 -0
- package/docs/releases/v0.4.0.md +62 -0
- package/docs/releases/v0.5.0.md +93 -0
- package/package.json +8 -15
- package/docs/integration/stdio-bridge.md +0 -670
- package/docs/integration/stdio-conformance.md +0 -51
- package/scripts/run-stdio-conformance.mjs +0 -698
package/README.md
CHANGED
|
@@ -1,444 +1,355 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Browser Pilot
|
|
2
2
|
|
|
3
|
-
Give
|
|
3
|
+
Give an AI Agent control of the user's real Chrome browser, including existing
|
|
4
|
+
logins, cookies, profiles, and extensions. No browser extension is required.
|
|
4
5
|
|
|
5
6
|
```bash
|
|
6
|
-
npm install -g browser-pilot-cli
|
|
7
|
+
npm install -g browser-pilot-cli@latest
|
|
8
|
+
bp --version
|
|
7
9
|
```
|
|
8
10
|
|
|
9
|
-
|
|
11
|
+
Native releases support Apple Silicon macOS, x64 Linux, and x64 Windows. Intel
|
|
12
|
+
Mac is not supported.
|
|
10
13
|
|
|
11
|
-
|
|
14
|
+
## Agent Setup
|
|
12
15
|
|
|
13
|
-
|
|
14
|
-
debugging. No command-line flags or restart are needed. Chrome may show its
|
|
15
|
-
separate Allow dialog when `bp connect` requests the actual connection.
|
|
16
|
+
### 1. Enable Chrome remote debugging
|
|
16
17
|
|
|
17
|
-
|
|
18
|
+
Open `chrome://inspect/#remote-debugging` in Chrome 144 or newer and enable
|
|
19
|
+
remote debugging. Browser Pilot does not use command-line debugging flags or a
|
|
20
|
+
Chrome extension.
|
|
18
21
|
|
|
19
|
-
|
|
22
|
+
Inspect installed browsers without requesting authorization:
|
|
20
23
|
|
|
21
24
|
```bash
|
|
22
25
|
bp browsers
|
|
23
26
|
```
|
|
24
27
|
|
|
25
|
-
|
|
26
|
-
a Chrome WebSocket and never display the Allow dialog.
|
|
28
|
+
Connect when browser work is needed:
|
|
27
29
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
One connection covers every live Chrome Profile on that endpoint. When several
|
|
32
|
-
Profiles are open, `bp connect` lists them without creating a Pilot window; use
|
|
33
|
-
`bp profiles` and `bp profile <index>`, or pass `--profile` to `bp open --new`.
|
|
30
|
+
```bash
|
|
31
|
+
bp connect
|
|
32
|
+
```
|
|
34
33
|
|
|
35
|
-
|
|
34
|
+
Chrome may display one Allow dialog for the browser connection. Browser
|
|
35
|
+
discovery and background-service startup are passive. Concurrent clients share
|
|
36
|
+
one pending connection request, so they do not create an authorization loop.
|
|
36
37
|
|
|
37
|
-
|
|
38
|
-
|
|
38
|
+
### 2. Install the Agent skill
|
|
39
|
+
|
|
40
|
+
Claude Code:
|
|
41
|
+
|
|
42
|
+
```text
|
|
39
43
|
/plugin marketplace add https://github.com/relixiaobo/browser-pilot.git
|
|
40
44
|
/plugin install browser-pilot@browser-pilot-marketplace
|
|
41
45
|
```
|
|
42
46
|
|
|
43
|
-
|
|
47
|
+
Codex, Cursor, or VS Code Copilot:
|
|
48
|
+
|
|
44
49
|
```bash
|
|
45
|
-
npx skills add relixiaobo/browser-pilot
|
|
50
|
+
npx skills add relixiaobo/browser-pilot --skill browser-pilot
|
|
46
51
|
```
|
|
47
52
|
|
|
48
|
-
|
|
53
|
+
OpenClaw or another Agent Skills-compatible host:
|
|
54
|
+
|
|
49
55
|
```bash
|
|
50
56
|
cp -r plugin/skills/browser-pilot ~/.agents/skills/
|
|
51
57
|
```
|
|
52
58
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
59
|
+
### 3. Ask for the browser task
|
|
60
|
+
|
|
61
|
+
- "Fill in the form I already opened."
|
|
62
|
+
- "Check my GitHub notifications."
|
|
63
|
+
- "Download the latest invoice and summarize it."
|
|
64
|
+
- "Open this report in a new tab and save it as PDF."
|
|
65
|
+
|
|
66
|
+
The Agent uses its existing shell tool to run `bp`. Browser Pilot does not add
|
|
67
|
+
native Agent tools, an MCP server, an SDK, or a persistent client protocol.
|
|
68
|
+
|
|
69
|
+
## Public Interface
|
|
70
|
+
|
|
71
|
+
Browser Pilot has one Agent integration surface:
|
|
72
|
+
|
|
73
|
+
```text
|
|
74
|
+
Agent -> Browser Pilot skill -> existing shell/command runner
|
|
75
|
+
-> bp CLI -> shared background service -> Chrome
|
|
56
76
|
```
|
|
57
77
|
|
|
58
|
-
|
|
78
|
+
The skill provides progressive disclosure and browser-specific operating
|
|
79
|
+
guidance. The CLI provides stable JSON, errors, state, files, waiting, and
|
|
80
|
+
recovery. The background service is private implementation detail shared by all
|
|
81
|
+
compatible CLI installations for the current OS user.
|
|
59
82
|
|
|
60
|
-
|
|
83
|
+
This same path supports both installation modes:
|
|
61
84
|
|
|
62
|
-
-
|
|
63
|
-
-
|
|
64
|
-
|
|
85
|
+
- An Agent installs `browser-pilot-cli` itself.
|
|
86
|
+
- A product bundles a pinned CLI and places its directory on the Agent command
|
|
87
|
+
environment's `PATH`.
|
|
65
88
|
|
|
66
|
-
|
|
89
|
+
Tenon, OpenClaw, Codex, and other shell-capable Agents need no Browser
|
|
90
|
+
Pilot-specific runtime integration. See
|
|
91
|
+
[the embedding reference](plugin/skills/browser-pilot/references/embedding.md).
|
|
67
92
|
|
|
68
|
-
##
|
|
93
|
+
## Browser Scope
|
|
69
94
|
|
|
70
|
-
|
|
71
|
-
- **Real login sessions** — Operates your actual browser profile. Cookies, extensions, logins all intact
|
|
72
|
-
- **Multi-Profile routing** — Lists user tabs across live Chrome Profiles and creates managed work only in a resolved Profile
|
|
73
|
-
- **CLI-native** — Any agent with bash access can use it. No MCP protocol, no SDK integration needed
|
|
74
|
-
- **Auto-snapshot** — Every action returns page state with numbered `[ref]` elements, so the agent always knows what's on screen
|
|
75
|
-
- **Adaptive page views** — Bounded read, text search, DOM metadata, page geometry, and annotated screenshots let an agent choose the smallest useful representation
|
|
76
|
-
- **Verified page primitives** — Scroll and dropdown tools return fresh state and typed evidence instead of requiring ad hoc JavaScript
|
|
77
|
-
- **Lightweight npm install** — About 200KB as an npm tarball. No bundled Chromium (unlike Playwright's 400MB+)
|
|
78
|
-
- **Rich editor support** — Works with contenteditable editors (Draft.js, ProseMirror, Quill, Slate) and Shadow DOM elements out of the box
|
|
95
|
+
`bp tabs` exposes all eligible page tabs on the authorized browser endpoint:
|
|
79
96
|
|
|
80
|
-
|
|
97
|
+
- Browser Pilot-created managed tabs;
|
|
98
|
+
- eligible popups from managed tabs;
|
|
99
|
+
- eligible tabs the user opened in any live Chrome profile.
|
|
81
100
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
| **Extension required** | No | No | No | No |
|
|
87
|
-
| **Element refs** | Numbered (accessibility tree) | Named refs (ARIA) | CSS selectors | Numbered (DOM) |
|
|
88
|
-
| **Auto-snapshot after action** | Yes | Yes | No | Yes |
|
|
89
|
-
| **Network interception** | Yes (block/mock/headers) | Yes | Yes | No |
|
|
90
|
-
| **Multi-browser** | Chromium-only | Chromium + Firefox + WebKit | Chromium-only | Chromium-only |
|
|
91
|
-
| **Dialog handling** | Explicit | Automatic | Manual | Automatic |
|
|
92
|
-
| **JSON output** | Default | MCP structured | MCP structured | Python objects |
|
|
93
|
-
| **File upload** | Auto-detect input | Yes | No | Yes |
|
|
101
|
+
The Agent can therefore fill a form or inspect a page the user already opened.
|
|
102
|
+
Browser-internal and extension-owned pages are excluded. Existing Chrome tab
|
|
103
|
+
groups do not limit tab control, whether expanded or collapsed, but Browser
|
|
104
|
+
Pilot cannot inspect or manage tab groups without an extension.
|
|
94
105
|
|
|
95
|
-
|
|
106
|
+
Browser Pilot does not implement task-intent permission prompts. Invoking the
|
|
107
|
+
tool gives the Agent the full eligible browser capability surface. An Agent
|
|
108
|
+
host may apply its own policy or approval UX, but that is outside the Browser
|
|
109
|
+
Pilot contract.
|
|
96
110
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
111
|
+
User-opened tabs remain open when Agent state is released. `bp close --all`
|
|
112
|
+
closes only Browser Pilot-created tabs. `bp close` explicitly closes the
|
|
113
|
+
selected tab, including a user tab.
|
|
114
|
+
|
|
115
|
+
## Chrome Profiles
|
|
116
|
+
|
|
117
|
+
One authorized browser connection can expose several live Chrome profiles.
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
bp profiles
|
|
121
|
+
bp profiles --identify
|
|
122
|
+
bp profile 2
|
|
123
|
+
bp open https://example.com --new --profile 2
|
|
108
124
|
```
|
|
109
125
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
126
|
+
`bp profiles` is passive and includes representative tabs. Use
|
|
127
|
+
`--identify` only when account-aware identity is needed; it briefly opens and
|
|
128
|
+
closes a visible `chrome://version` page per unidentified profile and accepts
|
|
129
|
+
identity only after exact profile-path verification. Browser Pilot reports
|
|
130
|
+
verified profile/account names rather than forcing the Agent to speak in
|
|
131
|
+
ambiguous "Profile 1" labels.
|
|
115
132
|
|
|
116
|
-
|
|
133
|
+
Profile selection controls where new managed tabs are created. It does not
|
|
134
|
+
grant or restrict access to existing eligible tabs.
|
|
117
135
|
|
|
118
|
-
|
|
119
|
-
same Agent-neutral, multi-client Browser Broker.
|
|
120
|
-
The approved architecture and execution plans are:
|
|
136
|
+
## Multi-Agent State
|
|
121
137
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
- [Browser capability and reliability plan](docs/plans/browser-capability-evolution.md)
|
|
125
|
-
- [Stdio bridge integration contract](docs/integration/stdio-bridge.md)
|
|
126
|
-
- [Stdio black-box conformance suite](docs/integration/stdio-conformance.md)
|
|
127
|
-
- [Reference consumer adapters](https://github.com/relixiaobo/browser-pilot/tree/main/examples/adapters),
|
|
128
|
-
including Tenon and OpenClaw lifecycle mappings
|
|
129
|
-
|
|
130
|
-
The public integration direction remains CLI-only: one-shot commands for direct
|
|
131
|
-
Agent use and a persistent `bridge --stdio` mode for products that embed the
|
|
132
|
-
official executable. Browser Pilot will not require an extension, Native SDK,
|
|
133
|
-
or MCP server. The repository's consumer adapters are source examples built
|
|
134
|
-
only on the stdio contract; they are intentionally excluded from the npm
|
|
135
|
-
package and do not create a second public library API.
|
|
136
|
-
|
|
137
|
-
The dedicated Pilot window remains the default managed tab set for independent
|
|
138
|
-
Agent work. The Broker architecture also includes all eligible user tabs in the
|
|
139
|
-
Agent's inventory without a separate grant step, so an Agent can operate a page
|
|
140
|
-
the user already opened. Invoking or exposing the tool is the authorization
|
|
141
|
-
boundary; products may apply their own approval UX or remove operations when
|
|
142
|
-
launching the bridge. Bulk cleanup remains limited to managed tabs, and user
|
|
143
|
-
tabs remain open when a session ends.
|
|
144
|
-
|
|
145
|
-
The `browser-pilot bridge --stdio` transport, Broker lifecycle, browser tool
|
|
146
|
-
dispatch, event replay, protected Artifacts, protocol 1.1 transport limit
|
|
147
|
-
negotiation, and protocol 1.2 Chrome Profile routing are implemented. Browser
|
|
148
|
-
disconnect and explicit reconnect handling, scoped
|
|
149
|
-
download Artifacts, Workspace resource isolation, and typed watchdog events for
|
|
150
|
-
stalled navigation, selected-frame detach, pending dialogs, and repeated
|
|
151
|
-
browser-observable no-progress actions are also implemented. Bounded page
|
|
152
|
-
search, DOM metadata lookup, scrolling, native/ARIA dropdown operations,
|
|
153
|
-
Observation page geometry, and viewport screenshot annotations adapt the most
|
|
154
|
-
useful browser-use inspection patterns to the same scoped Broker contract. An
|
|
155
|
-
internal managed-target janitor is the sole owner of the browser-level CDP
|
|
156
|
-
connection, proxies daemon traffic over private IPC, and closes only
|
|
157
|
-
Broker-created tabs and their managed popup descendants if the daemon exits or
|
|
158
|
-
crashes. It does not persist target IDs or close user tabs. Broker startup and
|
|
159
|
-
browser discovery are passive. Only `bp connect` or the versioned
|
|
160
|
-
`browser.connect` tool requests Chrome authorization; concurrent clients share
|
|
161
|
-
one in-flight request, and a failed or dropped connection is never retried by a
|
|
162
|
-
timer.
|
|
163
|
-
Compatible Agent products reuse one per-user Broker through protocol
|
|
164
|
-
negotiation. Protected shutdown cannot terminate live embedded clients, and an
|
|
165
|
-
incompatible product must explicitly select a separate `BROWSER_PILOT_HOME`.
|
|
166
|
-
Global npm installation, local npm/npx use, and product-bundled absolute-path
|
|
167
|
-
launches are covered by distribution black-box tests. Native self-contained
|
|
168
|
-
release artifacts use one executable for the public CLI and its private Broker
|
|
169
|
-
and janitor roles, with no runtime download or system-Node fallback.
|
|
170
|
-
|
|
171
|
-
## Installation and Embedding
|
|
172
|
-
|
|
173
|
-
Agent-managed installation remains supported:
|
|
138
|
+
Every independent Agent must use one stable key across all short-lived CLI
|
|
139
|
+
processes:
|
|
174
140
|
|
|
175
141
|
```bash
|
|
176
|
-
|
|
177
|
-
bp connect
|
|
142
|
+
export BROWSER_PILOT_CLIENT_KEY=product.install.agent-01
|
|
178
143
|
bp tabs
|
|
144
|
+
bp snapshot
|
|
179
145
|
```
|
|
180
146
|
|
|
181
|
-
|
|
182
|
-
|
|
147
|
+
Or pass `--client-key` before the command. The key preserves the Agent's
|
|
148
|
+
selected profile, tab, frame, refs, auth, network rules, downloads, and command
|
|
149
|
+
history. Different keys are isolated and cannot overwrite one another.
|
|
150
|
+
|
|
151
|
+
Browser Pilot keeps this state in background-service memory, not global
|
|
152
|
+
`state.json` or `refs.json` files. `bp disconnect` releases only the current
|
|
153
|
+
key. It stops the service only when no other Agent is using it and the invoking
|
|
154
|
+
executable owns that service process.
|
|
155
|
+
|
|
156
|
+
Compatible global, local, and product-bundled CLIs reuse the same per-user
|
|
157
|
+
service through protocol negotiation. Executable identity is required only for
|
|
158
|
+
stopping the whole service. Use a distinct absolute `BROWSER_PILOT_HOME` only
|
|
159
|
+
for a deliberately incompatible isolated installation.
|
|
160
|
+
|
|
161
|
+
## Core Workflow
|
|
183
162
|
|
|
184
163
|
```bash
|
|
185
|
-
|
|
186
|
-
|
|
164
|
+
bp tabs # find existing user or managed tabs
|
|
165
|
+
bp tab 2 # select by current one-based index
|
|
166
|
+
bp open https://example.com --new # create independent managed work
|
|
167
|
+
bp snapshot # get controls with numbered refs
|
|
168
|
+
bp read main --limit 10000 # get bounded readable content
|
|
169
|
+
bp search "invoice total" # find text with nearby context
|
|
170
|
+
bp click 3 # act on a fresh ref
|
|
171
|
+
bp type 5 "hello" --clear # replace input and verify readback
|
|
172
|
+
bp press Enter # press a key and return fresh state
|
|
187
173
|
```
|
|
188
174
|
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
npm form launches `node_modules/browser-pilot-cli/dist/cli.js` with the
|
|
193
|
-
product's pinned Node runtime. The native archive launches its single
|
|
194
|
-
`browser-pilot` executable and does not require Node to be installed.
|
|
195
|
-
|
|
196
|
-
Each native archive contains `manifest.json`, per-file `SHA256SUMS`, licenses,
|
|
197
|
-
and an adjacent archive checksum. The manifest reports the actual signature
|
|
198
|
-
kind: Developer ID or ad-hoc on macOS, Authenticode or unsigned on Windows, and
|
|
199
|
-
unsigned on Linux. Release automation never labels an artifact signed when its
|
|
200
|
-
signing credentials were unavailable.
|
|
201
|
-
|
|
202
|
-
All three forms use the same protocol and compatible per-user Broker. Use a
|
|
203
|
-
distinct absolute `BROWSER_PILOT_HOME` only when a product deliberately needs
|
|
204
|
-
an incompatible isolated Broker.
|
|
205
|
-
|
|
206
|
-
## Commands
|
|
207
|
-
|
|
208
|
-
### Core Loop
|
|
209
|
-
|
|
210
|
-
| Command | Returns | Description |
|
|
211
|
-
|---------|---------|-------------|
|
|
212
|
-
| `bp open <url>` | snapshot | Navigate to URL |
|
|
213
|
-
| `bp open <url> --new --profile <selector>` | snapshot | Create managed work in one live Chrome Profile |
|
|
214
|
-
| `bp snapshot` | snapshot | Get interactive elements |
|
|
215
|
-
| `bp read [selector]` | text | Get bounded readable page/region content |
|
|
216
|
-
| `bp search <text>` | matches | Find bounded visible text and nearby context |
|
|
217
|
-
| `bp find <selector>` | elements | Inspect bounded DOM metadata and requested attributes |
|
|
218
|
-
| `bp scroll [direction]` | snapshot | Scroll page/element/text and return fresh state |
|
|
219
|
-
| `bp dropdown <target>` | options | List native or exposed ARIA dropdown options |
|
|
220
|
-
| `bp select <target> <option>` | snapshot | Select and verify a dropdown option |
|
|
221
|
-
| `bp click <ref>` | snapshot | Click element by ref number (`--double`, `--right`) |
|
|
222
|
-
| `bp click --xy x,y` | snapshot | Click at viewport coordinates (canvas, maps) |
|
|
223
|
-
| `bp locate <selector>` | coords | Get element center x,y + size (for `click --xy`) |
|
|
224
|
-
| `bp type <ref> <text>` | snapshot | Type into element (`--clear`, `--submit`) |
|
|
225
|
-
| `bp keyboard <text>` | snapshot | Type via keyboard events (`--click`, `--clear`) |
|
|
226
|
-
| `bp press <key>` | snapshot | Press key (Enter, Escape, Control+a, Meta+c) |
|
|
227
|
-
| `bp eval [js]` | value | Run JavaScript (escape hatch for anything) |
|
|
228
|
-
|
|
229
|
-
### Utilities
|
|
230
|
-
|
|
231
|
-
| Command | Description |
|
|
232
|
-
|---------|-------------|
|
|
233
|
-
| `bp screenshot [file]` | Capture screenshot (`--full`, `--selector`, `--annotate [refs]`) |
|
|
234
|
-
| `bp pdf [file]` | Save page as PDF (`--landscape`) |
|
|
235
|
-
| `bp cookies [domain]` | View cookies (includes HttpOnly) |
|
|
236
|
-
|
|
237
|
-
### Edge Cases
|
|
238
|
-
|
|
239
|
-
| Command | Description |
|
|
240
|
-
|---------|-------------|
|
|
241
|
-
| `bp upload <filepath>` | Upload file (auto-finds `<input type="file">`) |
|
|
242
|
-
| `bp auth <user> <pass>` | Set HTTP Basic Auth credentials (`--clear`) |
|
|
243
|
-
| `bp frame [index]` | List or switch iframe context (0 = top) |
|
|
244
|
-
| `bp dialogs` | List pending JavaScript dialogs |
|
|
245
|
-
| `bp dialog <id> --accept\|--dismiss` | Explicitly respond to a dialog (`--prompt`) |
|
|
246
|
-
|
|
247
|
-
Dialogs remain pending until explicitly accepted or dismissed. One-shot CLI
|
|
248
|
-
dialogs are scoped to the compatibility Workspace and isolated from embedded
|
|
249
|
-
Broker clients.
|
|
250
|
-
|
|
251
|
-
Run `bp tabs` to list Pilot-managed tabs, their popups, and eligible tabs the
|
|
252
|
-
user opened elsewhere across all live Profiles in the same browser endpoint.
|
|
253
|
-
Each JSON entry includes its opaque `profileContextId`; `bp tab <n>` switches
|
|
254
|
-
control to any listed tab.
|
|
255
|
-
|
|
256
|
-
### Network
|
|
257
|
-
|
|
258
|
-
| Command | Description |
|
|
259
|
-
|---------|-------------|
|
|
260
|
-
| `bp net` | List recent requests (`--url`, `--method`, `--status`, `--type`) |
|
|
261
|
-
| `bp net show <id>` | Full request/response details (`--save <file>`) |
|
|
262
|
-
| `bp net block <pattern>` | Block requests matching URL pattern |
|
|
263
|
-
| `bp net mock <pattern>` | Mock responses (`--body`, `--file`) |
|
|
264
|
-
| `bp net headers <pattern> <header...>` | Add/override request headers |
|
|
265
|
-
| `bp net rules` | List active interception rules |
|
|
266
|
-
| `bp net remove [id]` | Remove rule(s) (`--all`) |
|
|
267
|
-
| `bp net clear` | Clear captured request log |
|
|
268
|
-
|
|
269
|
-
### Session
|
|
270
|
-
|
|
271
|
-
| Command | Description |
|
|
272
|
-
|---------|-------------|
|
|
273
|
-
| `bp connect` | Connect to Chrome; create a Pilot window immediately only when Profile routing is unambiguous |
|
|
274
|
-
| `bp disconnect` | Close the CLI Pilot window; stop the daemon when no embedded client is live |
|
|
275
|
-
| `bp profiles` | List live Chrome Profile contexts and representative tabs |
|
|
276
|
-
| `bp profile <index\|id\|label\|name>` | Select a Profile for subsequent managed tabs |
|
|
277
|
-
| `bp tabs` | List all controllable tabs in the current browser |
|
|
278
|
-
| `bp tab <n>` | Switch to any listed managed or user tab |
|
|
279
|
-
| `bp close` | Close the current tab (`--all` closes Pilot-managed tabs only) |
|
|
280
|
-
|
|
281
|
-
## Refs
|
|
282
|
-
|
|
283
|
-
Action commands return a snapshot of interactive elements, each with a `[ref]` number:
|
|
175
|
+
Action commands return fresh page state and typed browser-visible evidence.
|
|
176
|
+
Refs belong to the current tab, frame, and page state. Refresh them after
|
|
177
|
+
navigation, document changes, tab/frame switches, reconnect, or `stale_ref`.
|
|
284
178
|
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
[8] slider "Volume" ← <input type="range">
|
|
179
|
+
Use dedicated page primitives before JavaScript:
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
bp find ".result" --attributes href
|
|
183
|
+
bp scroll --to-text "Payment details"
|
|
184
|
+
bp dropdown 4
|
|
185
|
+
bp select 4 "United States"
|
|
186
|
+
bp screenshot page.png --annotate 1,3,8
|
|
294
187
|
```
|
|
295
188
|
|
|
296
|
-
|
|
189
|
+
`bp eval` remains an escape hatch when no dedicated command represents the
|
|
190
|
+
operation.
|
|
297
191
|
|
|
298
|
-
|
|
299
|
-
action. Elements inside Shadow DOM are included automatically. Snapshot JSON
|
|
300
|
-
may also include a `page` block with viewport/document size, scroll position,
|
|
301
|
-
remaining pixels, and scroll percentages.
|
|
192
|
+
## Async Work and Recovery
|
|
302
193
|
|
|
303
|
-
|
|
304
|
-
daemon's keyed compatibility Workspace and renewable Lease. They are never
|
|
305
|
-
written to `state.json` or `refs.json`; after Lease expiry the Agent must
|
|
306
|
-
observe again. `bp disconnect` explicitly clears the compatibility Workspace.
|
|
194
|
+
Wait for browser-visible state instead of using shell sleeps:
|
|
307
195
|
|
|
308
|
-
|
|
196
|
+
```bash
|
|
197
|
+
bp --timeout 30000 wait --url "*/complete"
|
|
198
|
+
bp --timeout 30000 wait --text "Payment received"
|
|
199
|
+
bp --timeout 30000 wait --selector ".result"
|
|
200
|
+
bp --timeout 30000 wait --dialog
|
|
201
|
+
bp --timeout 120000 wait --download
|
|
202
|
+
bp --timeout 30000 wait --popup
|
|
203
|
+
```
|
|
309
204
|
|
|
310
|
-
|
|
205
|
+
Inspect service and command state after a timeout, interrupted shell call, or
|
|
206
|
+
lost result:
|
|
311
207
|
|
|
312
|
-
```
|
|
313
|
-
|
|
208
|
+
```bash
|
|
209
|
+
bp status
|
|
210
|
+
bp commands --status accepted,dispatched,unknown_outcome
|
|
211
|
+
bp command <command-id>
|
|
212
|
+
bp cancel <command-id>
|
|
314
213
|
```
|
|
315
214
|
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
{"ok":false, "error":"Ref [99] not found.", "hint":"Run 'bp snapshot' to refresh element refs."}
|
|
319
|
-
```
|
|
215
|
+
Attach a stable request identity before the first attempt when the caller may
|
|
216
|
+
need to recover the same intended operation:
|
|
320
217
|
|
|
321
|
-
|
|
218
|
+
```bash
|
|
219
|
+
bp --request-id task-42-submit click 7
|
|
220
|
+
```
|
|
322
221
|
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
browser receives the file; the source file is never removed.
|
|
222
|
+
Reuse that ID only for recovery of the same intended call. Use a new ID for a
|
|
223
|
+
new action. Browser Pilot deduplicates stable requests and attempts
|
|
224
|
+
cancellation on CLI `SIGINT`/`SIGTERM`, but a dispatched mutation can still
|
|
225
|
+
have an `unknown_outcome`. Inspect the current page before retrying.
|
|
328
226
|
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
`--annotate 1,3,8` to limit it. Annotation does not inject elements into the
|
|
332
|
-
page, runs in an isolated JavaScript world, and cannot be combined with full-page
|
|
333
|
-
or selector capture.
|
|
227
|
+
Machine errors always include stable `code` and `retryable` fields. Branch on
|
|
228
|
+
`code`, not English text.
|
|
334
229
|
|
|
335
|
-
##
|
|
230
|
+
## Files, Screenshots, PDFs, and Downloads
|
|
336
231
|
|
|
337
|
-
|
|
232
|
+
Use explicit task-owned paths or set a default directory:
|
|
338
233
|
|
|
339
234
|
```bash
|
|
340
|
-
|
|
341
|
-
bp
|
|
342
|
-
bp
|
|
343
|
-
bp
|
|
344
|
-
bp
|
|
345
|
-
bp
|
|
346
|
-
|
|
235
|
+
export BROWSER_PILOT_OUTPUT_DIR=/absolute/task/output
|
|
236
|
+
bp screenshot
|
|
237
|
+
bp screenshot full.png --full
|
|
238
|
+
bp pdf report.pdf --landscape
|
|
239
|
+
bp downloads
|
|
240
|
+
bp download 1 invoice.pdf
|
|
241
|
+
bp net show 12 --save response.json
|
|
347
242
|
```
|
|
348
243
|
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
244
|
+
File results return an absolute `file` path, MIME type, byte size, and image
|
|
245
|
+
dimensions when available. An Agent can open that path with its normal image or
|
|
246
|
+
file-reading capability. CLI output remains bounded JSON and never embeds large
|
|
247
|
+
binary payloads.
|
|
352
248
|
|
|
353
|
-
|
|
249
|
+
Browser Pilot copies uploads and completed downloads into protected temporary
|
|
250
|
+
storage while processing them. It never removes the source upload or Chrome's
|
|
251
|
+
original download.
|
|
354
252
|
|
|
355
|
-
|
|
253
|
+
## Command Groups
|
|
356
254
|
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
255
|
+
| Area | Commands |
|
|
256
|
+
|------|----------|
|
|
257
|
+
| Setup and state | `browsers`, `connect`, `status`, `disconnect` |
|
|
258
|
+
| Profiles and tabs | `profiles`, `profile`, `tabs`, `tab`, `open`, `close` |
|
|
259
|
+
| Observation | `snapshot`, `read`, `search`, `find`, `locate` |
|
|
260
|
+
| Interaction | `click`, `type`, `keyboard`, `press`, `scroll`, `dropdown`, `select`, `eval` |
|
|
261
|
+
| Async and recovery | `wait`, `commands`, `command`, `cancel` |
|
|
262
|
+
| Page context | `frame`, `dialogs`, `dialog` |
|
|
263
|
+
| Files and media | `upload`, `downloads`, `download`, `screenshot`, `pdf` |
|
|
264
|
+
| Browser data | `cookies`, `auth`, `net` |
|
|
362
265
|
|
|
363
|
-
|
|
266
|
+
Run `bp --help` or `bp <command> --help` for exact options.
|
|
364
267
|
|
|
365
|
-
|
|
268
|
+
## How It Works
|
|
366
269
|
|
|
367
|
-
```
|
|
368
|
-
bp
|
|
270
|
+
```text
|
|
271
|
+
short-lived bp CLI processes
|
|
272
|
+
|
|
|
273
|
+
| private local HTTP/Unix socket
|
|
274
|
+
v
|
|
275
|
+
per-user Browser Pilot service
|
|
276
|
+
|
|
|
277
|
+
| one supervised CDP WebSocket per browser endpoint
|
|
278
|
+
v
|
|
279
|
+
Chrome profiles and eligible tabs
|
|
369
280
|
```
|
|
370
281
|
|
|
371
|
-
|
|
282
|
+
The service serializes actions per physical tab while allowing independent
|
|
283
|
+
tabs to progress concurrently. Each Agent receives isolated logical state and
|
|
284
|
+
opaque tab/ref addresses. The same physical user tab can be controlled by only
|
|
285
|
+
one Agent at a time.
|
|
372
286
|
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
bp keyboard "bold text"
|
|
377
|
-
bp click --xy 400,300 # click canvas area
|
|
378
|
-
```
|
|
287
|
+
An internal connection supervisor owns Browser Pilot-created tabs so it can
|
|
288
|
+
clean them up after a service crash without touching user-opened tabs. Browser
|
|
289
|
+
disconnect invalidates old tab, frame, and ref identities before reconnect.
|
|
379
290
|
|
|
380
|
-
|
|
291
|
+
See:
|
|
381
292
|
|
|
382
|
-
|
|
293
|
+
- [Current architecture](docs/architecture/browser-pilot-platform-spec.md)
|
|
294
|
+
- [Universal Agent integration plan](docs/plans/universal-agent-integration.md)
|
|
295
|
+
- [Browser capability evolution](docs/plans/browser-capability-evolution.md)
|
|
296
|
+
- [Chrome profile routing](docs/plans/profile-context-routing.md)
|
|
383
297
|
|
|
384
|
-
|
|
385
|
-
bp eval 'document.querySelector("select").value = "opt2"; document.querySelector("select").dispatchEvent(new Event("change",{bubbles:true}))'
|
|
386
|
-
```
|
|
298
|
+
## Comparison
|
|
387
299
|
|
|
388
|
-
|
|
300
|
+
| | Browser Pilot | browser-use | Playwright MCP | Chrome DevTools MCP |
|
|
301
|
+
|---|---|---|---|---|
|
|
302
|
+
| Agent interface | Skill + CLI | Python library/Agent runtime | MCP tools | MCP tools |
|
|
303
|
+
| User's signed-in Chrome | Yes | Yes | Usually isolated browser | Depends on launch mode |
|
|
304
|
+
| Browser extension | No | No | No | No |
|
|
305
|
+
| Eligible user tabs | Yes | Browser-session dependent | Browser-session dependent | Yes |
|
|
306
|
+
| Progressive Agent guidance | Skill references | Library/Agent prompts | Tool descriptions | Tool descriptions |
|
|
307
|
+
| Action verification | Typed evidence + fresh state | State/history checks | Fresh snapshots | Manual inspection |
|
|
308
|
+
| Async recovery | Wait + command status + request IDs | Agent/controller lifecycle | Host/MCP lifecycle | Host/MCP lifecycle |
|
|
309
|
+
| Local media results | Paths + metadata | Library objects/paths | MCP image/resource results | MCP results |
|
|
310
|
+
| Network interception | Observe, block, mock, headers | Limited by controller | Supported | Supported |
|
|
311
|
+
|
|
312
|
+
Browser Pilot adopts several strong browser-use ideas without adopting its
|
|
313
|
+
Python Agent runtime: adaptive DOM/accessibility observation, fresh state after
|
|
314
|
+
actions, input readback, obstruction checks, stale-state invalidation,
|
|
315
|
+
watchdogs, and bounded outputs. The Agent interface remains the CLI.
|
|
316
|
+
|
|
317
|
+
## Product Embedding
|
|
318
|
+
|
|
319
|
+
Bundle the native CLI or `browser-pilot-cli` plus a pinned Node runtime. Verify
|
|
320
|
+
the versioned release index and checksums. Put the bundled command directory on
|
|
321
|
+
the Agent's `PATH`, install the complete skill, and inject:
|
|
322
|
+
|
|
323
|
+
```text
|
|
324
|
+
BROWSER_PILOT_CLIENT_KEY=<stable logical Agent identity>
|
|
325
|
+
BROWSER_PILOT_OUTPUT_DIR=<absolute task-owned directory>
|
|
326
|
+
```
|
|
389
327
|
|
|
390
|
-
|
|
328
|
+
Do not map every CLI command into a native tool. The Agent already has a shell
|
|
329
|
+
tool, the skill controls context growth, and `bp --help` provides runtime
|
|
330
|
+
discovery. Agent-managed and product-bundled installations therefore exercise
|
|
331
|
+
the exact same implementation and guidance.
|
|
391
332
|
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
bp net --url "*api*" --method POST # filter by URL and method
|
|
396
|
-
bp net show 3 # full details + response body
|
|
397
|
-
|
|
398
|
-
# Block requests
|
|
399
|
-
bp net block "*tracking*" # block analytics/tracking
|
|
400
|
-
bp net block "*ads*"
|
|
401
|
-
|
|
402
|
-
# Mock API responses
|
|
403
|
-
bp net mock "*api/data*" --body '{"ok":true}'
|
|
404
|
-
bp net mock "*api/users*" --file mock.json
|
|
405
|
-
|
|
406
|
-
# Override request headers
|
|
407
|
-
bp net headers "*api*" "Authorization:Bearer test123"
|
|
408
|
-
|
|
409
|
-
# Manage rules
|
|
410
|
-
bp net rules # list active rules
|
|
411
|
-
bp net remove <id-from-bp-net-rules> # rule IDs are opaque
|
|
412
|
-
bp net remove --all # clear all rules
|
|
413
|
-
bp net clear # clear captured request log
|
|
414
|
-
```
|
|
333
|
+
Each release ships a plugin archive, native archives, checksums, and a release
|
|
334
|
+
index binding the CLI version, compatible skill range, protocol range, and
|
|
335
|
+
supported assets. No `darwin-x64` artifact is produced.
|
|
415
336
|
|
|
416
337
|
## Testing
|
|
417
338
|
|
|
418
|
-
Local release tests are deterministic. Real-site checks run separately as
|
|
419
|
-
non-blocking canaries so a third-party outage cannot fail the release gate:
|
|
420
|
-
|
|
421
339
|
```bash
|
|
422
|
-
npm test
|
|
423
|
-
npm
|
|
424
|
-
npm run test:
|
|
425
|
-
npm run test:distribution
|
|
426
|
-
npm run test:canary
|
|
427
|
-
npm run test:canary:strict
|
|
428
|
-
npm run test:integration # compatibility alias for strict canaries
|
|
429
|
-
npm run test:all # release gates plus non-blocking canaries
|
|
340
|
+
npm run test:unit # TypeScript build and all Node tests
|
|
341
|
+
npm test # unit tests plus local Playwright gates
|
|
342
|
+
npm run test:capabilities # isolated-Chrome quantitative capability gate
|
|
343
|
+
npm run test:distribution # packed global/local/bundled CLI verification
|
|
344
|
+
npm run test:canary # non-blocking real-site drift report
|
|
345
|
+
npm run test:canary:strict
|
|
430
346
|
```
|
|
431
347
|
|
|
432
|
-
Automated browser
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
files. Tests fail closed if that isolation is absent; they never attach to the
|
|
436
|
-
user's Chrome by default. A maintainer can explicitly opt into a destructive
|
|
437
|
-
manual compatibility run against the user's browser by setting
|
|
438
|
-
`BROWSER_PILOT_TEST_USER_CHROME=1`.
|
|
348
|
+
Automated browser tests use isolated temporary Chrome profiles and Broker homes.
|
|
349
|
+
They never attach to the user's Chrome unless a maintainer explicitly opts into
|
|
350
|
+
a controlled manual test.
|
|
439
351
|
|
|
440
|
-
|
|
441
|
-
SEA-capable Node 22.17.0 runtime:
|
|
352
|
+
Build and verify a native artifact with an SEA-capable Node runtime:
|
|
442
353
|
|
|
443
354
|
```bash
|
|
444
355
|
BROWSER_PILOT_SEA_NODE=/absolute/path/to/node npm run build:standalone
|
|
@@ -446,16 +357,21 @@ npm run verify:standalone
|
|
|
446
357
|
npm run package:standalone
|
|
447
358
|
```
|
|
448
359
|
|
|
449
|
-
The
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
360
|
+
The root `package.json` version is the release source of truth. The npm version
|
|
361
|
+
lifecycle synchronizes the lockfile, plugin manifests, skill compatibility
|
|
362
|
+
range, and marketplace entry:
|
|
363
|
+
|
|
364
|
+
```bash
|
|
365
|
+
npm version "$BROWSER_PILOT_RELEASE_VERSION" --no-git-tag-version
|
|
366
|
+
npm run release:check-version
|
|
367
|
+
```
|
|
453
368
|
|
|
454
369
|
## Requirements
|
|
455
370
|
|
|
456
|
-
- Chrome 144
|
|
457
|
-
- Node.js
|
|
458
|
-
- Remote debugging enabled
|
|
371
|
+
- Chrome 144+, Edge, Brave, or another supported Chromium browser
|
|
372
|
+
- Node.js 18 or newer for the npm distribution
|
|
373
|
+
- Remote debugging enabled at `chrome://inspect/#remote-debugging`
|
|
374
|
+
- Apple Silicon macOS, x64 Linux, or x64 Windows for native releases
|
|
459
375
|
|
|
460
376
|
## License
|
|
461
377
|
|