browser-pilot-cli 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,488 +1,355 @@
1
- # browser-pilot
1
+ # Browser Pilot
2
2
 
3
- Give your AI agent control of your real browser — with your logins, cookies, and extensions intact. No extension needed.
3
+ Give an AI Agent control of the user's real Chrome browser, including existing
4
+ logins, cookies, profiles, and extensions. No browser extension is required.
4
5
 
5
6
  ```bash
6
7
  npm install -g browser-pilot-cli@latest
7
8
  bp --version
8
9
  ```
9
10
 
10
- Each skill/plugin release declares a supported CLI range and the exact version
11
- it was tested with. Newer CLI releases remain usable within that range;
12
- embedded products should pin the tested version for reproducible builds.
13
- The native release targets Apple Silicon macOS, x64 Linux, and x64 Windows;
14
- Intel Mac is not supported.
11
+ Native releases support Apple Silicon macOS, x64 Linux, and x64 Windows. Intel
12
+ Mac is not supported.
15
13
 
16
14
  ## Agent Setup
17
15
 
18
- ### 1. Enable Chrome remote debugging (one-time)
16
+ ### 1. Enable Chrome remote debugging
19
17
 
20
- Open `chrome://inspect/#remote-debugging` in Chrome (144+) and enable remote
21
- debugging. No command-line flags or restart are needed. Chrome may show its
22
- separate Allow dialog when `bp connect` requests the actual connection.
18
+ Open `chrome://inspect/#remote-debugging` in Chrome 144 or newer and enable
19
+ remote debugging. Browser Pilot does not use command-line debugging flags or a
20
+ Chrome extension.
23
21
 
24
- > Chrome 136 disabled the old `--remote-debugging-port` flag for security. Chrome 144 introduced this new UI toggle as the replacement — browser-pilot uses this.
25
-
26
- Check every installed supported browser and its exact setup state with:
22
+ Inspect installed browsers without requesting authorization:
27
23
 
28
24
  ```bash
29
25
  bp browsers
30
26
  ```
31
27
 
32
- Discovery is passive: `bp browsers`, bridge startup, and `initialize` never open
33
- a Chrome WebSocket and never display the Allow dialog.
28
+ Connect when browser work is needed:
29
+
30
+ ```bash
31
+ bp connect
32
+ ```
33
+
34
+ Chrome may display one Allow dialog for the browser connection. Browser
35
+ discovery and background-service startup are passive. Concurrent clients share
36
+ one pending connection request, so they do not create an authorization loop.
34
37
 
35
- Use `bp connect --browser <id|product|channel>` for an explicit choice. Product
36
- embedders can pass the same selector to `browser-pilot bridge --stdio
37
- --browser <selector>`; otherwise selection follows a stable platform order.
38
- One connection covers every live Chrome Profile on that endpoint. When several
39
- Profiles are open, `bp connect` lists them without creating a Pilot window. Use
40
- the passive `bp profiles` inventory first. Run `bp profiles --identify` only
41
- when account-aware names are needed; it briefly opens and closes one visible
42
- `chrome://version` page per unidentified Profile. Profile and tab indexes are
43
- one-based. Select with `bp profile <index>`, or pass `--profile` to
44
- `bp open --new`.
38
+ ### 2. Install the Agent skill
45
39
 
46
- ### 2. Install the plugin for your agent
40
+ Claude Code:
47
41
 
48
- **Claude Code:**
49
- ```
42
+ ```text
50
43
  /plugin marketplace add https://github.com/relixiaobo/browser-pilot.git
51
44
  /plugin install browser-pilot@browser-pilot-marketplace
52
45
  ```
53
46
 
54
- **Codex CLI:**
47
+ Codex, Cursor, or VS Code Copilot:
48
+
55
49
  ```bash
56
50
  npx skills add relixiaobo/browser-pilot --skill browser-pilot
57
51
  ```
58
52
 
59
- **OpenClaw:**
53
+ OpenClaw or another Agent Skills-compatible host:
54
+
60
55
  ```bash
61
56
  cp -r plugin/skills/browser-pilot ~/.agents/skills/
62
57
  ```
63
58
 
64
- **Cursor / VS Code Copilot:**
65
- ```bash
66
- npx skills add relixiaobo/browser-pilot --skill browser-pilot
67
- ```
59
+ ### 3. Ask for the browser task
68
60
 
69
- ### 3. Use it
61
+ - "Fill in the form I already opened."
62
+ - "Check my GitHub notifications."
63
+ - "Download the latest invoice and summarize it."
64
+ - "Open this report in a new tab and save it as PDF."
70
65
 
71
- Just tell your agent what you want to do:
66
+ The Agent uses its existing shell tool to run `bp`. Browser Pilot does not add
67
+ native Agent tools, an MCP server, an SDK, or a persistent client protocol.
72
68
 
73
- - "Open GitHub and check my notifications"
74
- - "Go to Hacker News and summarize the top 5 posts"
75
- - "Fill out the form on this page"
69
+ ## Public Interface
76
70
 
77
- The agent will use `bp` commands automatically. Your real login sessions are preserved — no need to re-authenticate.
71
+ Browser Pilot has one Agent integration surface:
78
72
 
79
- ## Why browser-pilot?
73
+ ```text
74
+ Agent -> Browser Pilot skill -> existing shell/command runner
75
+ -> bp CLI -> shared background service -> Chrome
76
+ ```
80
77
 
81
- - **No extension required** Uses Chrome 144's native remote debugging toggle, not the Extension Debugger API
82
- - **Real login sessions** Operates your actual browser profile. Cookies, extensions, logins all intact
83
- - **Multi-Profile routing** Lists user tabs across live Chrome Profiles, explicitly verifies account-aware Profile identity, and creates managed work only in a resolved Profile
84
- - **CLI-native** Any agent with bash access can use it. No MCP protocol, no SDK integration needed
85
- - **Auto-snapshot** — Every action returns page state with numbered `[ref]` elements, so the agent always knows what's on screen
86
- - **Adaptive page views** — Bounded read, text search, DOM metadata, page geometry, and annotated screenshots let an agent choose the smallest useful representation
87
- - **Verified page primitives** — Scroll and dropdown tools return fresh state and typed evidence instead of requiring ad hoc JavaScript
88
- - **Lightweight npm install** — About 200KB as an npm tarball. No bundled Chromium (unlike Playwright's 400MB+)
89
- - **Rich editor support** — Works with contenteditable editors (Draft.js, ProseMirror, Quill, Slate) and Shadow DOM elements out of the box
78
+ The skill provides progressive disclosure and browser-specific operating
79
+ guidance. The CLI provides stable JSON, errors, state, files, waiting, and
80
+ recovery. The background service is private implementation detail shared by all
81
+ compatible CLI installations for the current OS user.
90
82
 
91
- ## Comparison
83
+ This same path supports both installation modes:
92
84
 
93
- | | browser-pilot | Playwright MCP | Chrome DevTools MCP | browser-use |
94
- |---|---|---|---|---|
95
- | **Interface** | CLI (bash) | MCP protocol | MCP protocol | Python SDK |
96
- | **Login session reuse** | Yes | No | Depends | Yes |
97
- | **Extension required** | No | No | No | No |
98
- | **Element refs** | Numbered (accessibility tree) | Named refs (ARIA) | CSS selectors | Numbered (DOM) |
99
- | **Auto-snapshot after action** | Yes | Yes | No | Yes |
100
- | **Network interception** | Yes (block/mock/headers) | Yes | Yes | No |
101
- | **Multi-browser** | Chromium-only | Chromium + Firefox + WebKit | Chromium-only | Chromium-only |
102
- | **Dialog handling** | Explicit | Automatic | Manual | Automatic |
103
- | **JSON output** | Default | MCP structured | MCP structured | Python objects |
104
- | **File upload** | Auto-detect input | Yes | No | Yes |
85
+ - An Agent installs `browser-pilot-cli` itself.
86
+ - A product bundles a pinned CLI and places its directory on the Agent command
87
+ environment's `PATH`.
105
88
 
106
- ## How It Works
89
+ Tenon, OpenClaw, Codex, and other shell-capable Agents need no Browser
90
+ Pilot-specific runtime integration. See
91
+ [the embedding reference](plugin/skills/browser-pilot/references/embedding.md).
107
92
 
108
- ```
109
- Agent (bash tool)
110
- │ bp open / bp click / bp eval ...
111
-
112
- CLI Process ──── HTTP/Unix Socket ──── Daemon Process (persistent)
113
-
114
- │ WebSocket (CDP)
115
-
116
- Chrome (your browser, your Profiles)
117
- ├── Eligible user tabs (selectable)
118
- └── Managed windows (Broker-owned)
119
- ```
93
+ ## Browser Scope
120
94
 
121
- The Broker maintains isolated state for each Agent. A private connection
122
- supervisor owns the Broker's single browser-level CDP WebSocket and proxies it
123
- to the daemon. It also owns only Browser Pilot-created tabs so it can clean them
124
- after a Broker crash without touching user-opened tabs. Controller identity is
125
- returned to the embedding Host; Browser Pilot never injects a status border or
126
- other control UI into the page DOM.
95
+ `bp tabs` exposes all eligible page tabs on the authorized browser endpoint:
127
96
 
128
- ## Platform Evolution
97
+ - Browser Pilot-created managed tabs;
98
+ - eligible popups from managed tabs;
99
+ - eligible tabs the user opened in any live Chrome profile.
129
100
 
130
- Browser Pilot now routes direct `bp` commands and embedded clients through the
131
- same Agent-neutral, multi-client Browser Broker.
132
- The approved architecture and execution plans are:
101
+ The Agent can therefore fill a form or inspect a page the user already opened.
102
+ Browser-internal and extension-owned pages are excluded. Existing Chrome tab
103
+ groups do not limit tab control, whether expanded or collapsed, but Browser
104
+ Pilot cannot inspect or manage tab groups without an extension.
133
105
 
134
- - [Platform specification](docs/architecture/browser-pilot-platform-spec.md)
135
- - [Universal Agent integration plan](docs/plans/universal-agent-integration.md)
136
- - [Browser capability and reliability plan](docs/plans/browser-capability-evolution.md)
137
- - [Stdio bridge integration contract](docs/integration/stdio-bridge.md)
138
- - [Stdio black-box conformance suite](docs/integration/stdio-conformance.md)
139
- - [Reference consumer adapters](https://github.com/relixiaobo/browser-pilot/tree/main/examples/adapters),
140
- including Tenon and OpenClaw lifecycle mappings
141
-
142
- The public integration direction remains CLI-only: one-shot commands for direct
143
- Agent use and a persistent `bridge --stdio` mode for products that embed the
144
- official executable. Browser Pilot will not require an extension, Native SDK,
145
- or MCP server. The repository's consumer adapters are source examples built
146
- only on the stdio contract; they are intentionally excluded from the npm
147
- package and do not create a second public library API.
148
-
149
- The dedicated Pilot window remains the default managed tab set for independent
150
- Agent work. The Broker architecture also includes all eligible user tabs in the
151
- Agent's inventory without a separate grant step, so an Agent can operate a page
152
- the user already opened. Invoking or exposing the tool is the authorization
153
- boundary; products may apply their own approval UX or remove operations when
154
- launching the bridge. Bulk cleanup remains limited to managed tabs, and user
155
- tabs remain open when a session ends.
156
-
157
- The `browser-pilot bridge --stdio` transport, Broker lifecycle, browser tool
158
- dispatch, event replay, protected Artifacts, protocol 1.1 transport limit
159
- negotiation, protocol 1.2 Chrome Profile routing, and protocol 1.3 verified
160
- Profile identity and unambiguous result vocabulary are implemented. Browser
161
- disconnect and explicit reconnect handling, scoped
162
- download Artifacts, Workspace resource isolation, and typed watchdog events for
163
- stalled navigation, selected-frame detach, pending dialogs, and repeated
164
- browser-observable no-progress actions are also implemented. Bounded page
165
- search, DOM metadata lookup, scrolling, native/ARIA dropdown operations,
166
- Observation page geometry, and viewport screenshot annotations adapt the most
167
- useful browser-use inspection patterns to the same scoped Broker contract. An
168
- internal managed-target janitor is the sole owner of the browser-level CDP
169
- connection, proxies daemon traffic over private IPC, and closes only
170
- Broker-created tabs and their managed popup descendants if the daemon exits or
171
- crashes. It does not persist target IDs or close user tabs. Broker startup and
172
- browser discovery are passive. Only `bp connect` or the versioned
173
- `browser.connect` tool requests Chrome authorization; concurrent clients share
174
- one in-flight request, and a failed or dropped connection is never retried by a
175
- timer.
176
- Compatible Agent products reuse one per-user Broker through protocol
177
- negotiation. Protected shutdown cannot terminate live embedded clients or
178
- another CLI Agent's active Lease, and an
179
- incompatible product must explicitly select a separate `BROWSER_PILOT_HOME`.
180
- Global npm installation, local npm/npx use, and product-bundled absolute-path
181
- launches are covered by distribution black-box tests. Native self-contained
182
- release artifacts use one executable for the public CLI and its private Broker
183
- and janitor roles, with no runtime download or system-Node fallback.
184
-
185
- ## Installation and Embedding
186
-
187
- Agent-managed installation remains supported:
106
+ Browser Pilot does not implement task-intent permission prompts. Invoking the
107
+ tool gives the Agent the full eligible browser capability surface. An Agent
108
+ host may apply its own policy or approval UX, but that is outside the Browser
109
+ Pilot contract.
188
110
 
189
- ```bash
190
- npm install -g browser-pilot-cli@latest
191
- bp --version
192
- bp connect
193
- bp tabs
194
- ```
111
+ User-opened tabs remain open when Agent state is released. `bp close --all`
112
+ closes only Browser Pilot-created tabs. `bp close` explicitly closes the
113
+ selected tab, including a user tab.
195
114
 
196
- A project can pin Browser Pilot locally and run the same CLI without a global
197
- installation:
115
+ ## Chrome Profiles
116
+
117
+ One authorized browser connection can expose several live Chrome profiles.
198
118
 
199
119
  ```bash
200
- npm install --save-exact browser-pilot-cli@latest
201
- npx --no-install browser-pilot tabs
120
+ bp profiles
121
+ bp profiles --identify
122
+ bp profile 2
123
+ bp open https://example.com --new --profile 2
202
124
  ```
203
125
 
204
- Agent products should pin either the npm package or a native archive from the
205
- GitHub release. Launch an absolute path directly with `bridge --stdio`; do not
206
- invoke a shell, import `src/*`, or depend on Broker locator/socket files. The
207
- npm form launches `node_modules/browser-pilot-cli/dist/cli.js` with the
208
- product's pinned Node runtime. The native archive launches its single
209
- `browser-pilot` executable and does not require Node to be installed.
210
-
211
- Each native archive contains `manifest.json`, per-file `SHA256SUMS`, licenses,
212
- and an adjacent archive checksum. The manifest reports the actual signature
213
- kind: Developer ID or ad-hoc on macOS, Authenticode or unsigned on Windows, and
214
- unsigned on Linux. Release automation never labels an artifact signed when its
215
- signing credentials were unavailable.
216
-
217
- The same GitHub release also contains
218
- `browser-pilot-plugin-<version>.tgz`, its checksum, and a versioned release index
219
- that binds the npm version, protocol range, compatible skill/plugin range,
220
- tested CLI version, and the three supported native assets. Consumers should
221
- verify that index before bundling. No `darwin-x64` asset is produced.
222
-
223
- All three forms use the same protocol and compatible per-user Broker. Use a
224
- distinct absolute `BROWSER_PILOT_HOME` only when a product deliberately needs
225
- an incompatible isolated Broker.
226
-
227
- Products embedding `bridge --stdio` already receive isolated Connections,
228
- Workspaces, and Leases. When several independent Agents instead invoke the
229
- one-shot CLI directly, assign each integration a distinct stable namespace:
126
+ `bp profiles` is passive and includes representative tabs. Use
127
+ `--identify` only when account-aware identity is needed; it briefly opens and
128
+ closes a visible `chrome://version` page per unidentified profile and accepts
129
+ identity only after exact profile-path verification. Browser Pilot reports
130
+ verified profile/account names rather than forcing the Agent to speak in
131
+ ambiguous "Profile 1" labels.
132
+
133
+ Profile selection controls where new managed tabs are created. It does not
134
+ grant or restrict access to existing eligible tabs.
135
+
136
+ ## Multi-Agent State
137
+
138
+ Every independent Agent must use one stable key across all short-lived CLI
139
+ processes:
230
140
 
231
141
  ```bash
232
- BROWSER_PILOT_CLIENT_KEY=agent.product.install-01 bp tabs
233
- bp --client-key agent.product.install-01 snapshot
142
+ export BROWSER_PILOT_CLIENT_KEY=product.install.agent-01
143
+ bp tabs
144
+ bp snapshot
234
145
  ```
235
146
 
236
- Repeated commands with the same key reuse that Agent's selected target, frame,
237
- refs, auth, and network state. Different keys cannot overwrite one another.
238
- The default key remains `browser-pilot-cli` for a single interactive Agent.
239
-
240
- ## Commands
241
-
242
- ### Core Loop
243
-
244
- | Command | Returns | Description |
245
- |---------|---------|-------------|
246
- | `bp open <url>` | snapshot | Navigate to URL |
247
- | `bp open <url> --new --profile <selector>` | snapshot | Create managed work in one live Chrome Profile |
248
- | `bp snapshot` | snapshot | Get interactive elements |
249
- | `bp read [selector]` | text | Get bounded readable page/region content |
250
- | `bp search <text>` | matches | Find bounded visible text and nearby context |
251
- | `bp find <selector>` | elements | Inspect bounded DOM metadata and requested attributes |
252
- | `bp scroll [direction]` | snapshot | Scroll page/element/text and return fresh state |
253
- | `bp dropdown <target>` | options | List native or exposed ARIA dropdown options |
254
- | `bp select <target> <option>` | snapshot | Select and verify a dropdown option |
255
- | `bp click <ref>` | snapshot | Click element by ref number (`--double`, `--right`) |
256
- | `bp click --xy x,y` | snapshot | Click at viewport coordinates (canvas, maps) |
257
- | `bp locate <selector>` | coords | Get element center x,y + size (for `click --xy`) |
258
- | `bp type <ref> <text>` | snapshot | Type into element (`--clear`, `--submit`) |
259
- | `bp keyboard <text>` | snapshot | Type via keyboard events (`--click`, `--clear`) |
260
- | `bp press <key>` | snapshot | Press key (Enter, Escape, Control+a, Meta+c) |
261
- | `bp eval [js]` | value | Run JavaScript (escape hatch for anything) |
262
-
263
- ### Utilities
264
-
265
- | Command | Description |
266
- |---------|-------------|
267
- | `bp screenshot [file]` | Capture screenshot (`--full`, `--selector`, `--annotate [refs]`) |
268
- | `bp pdf [file]` | Save page as PDF (`--landscape`) |
269
- | `bp cookies [domain]` | View cookies (includes HttpOnly) |
270
-
271
- ### Edge Cases
272
-
273
- | Command | Description |
274
- |---------|-------------|
275
- | `bp upload <filepath>` | Upload file (auto-finds `<input type="file">`) |
276
- | `bp auth <user> <pass>` | Set HTTP Basic Auth credentials (`--clear`) |
277
- | `bp frame [index]` | List or switch iframe context (0 = top) |
278
- | `bp dialogs` | List pending JavaScript dialogs |
279
- | `bp dialog <id> --accept\|--dismiss` | Explicitly respond to a dialog (`--prompt`) |
280
-
281
- Dialogs remain pending until explicitly accepted or dismissed. One-shot CLI
282
- dialogs are scoped to the compatibility Workspace and isolated from embedded
283
- Broker clients.
284
-
285
- Run `bp tabs` to list Pilot-managed tabs, their popups, and eligible tabs the
286
- user opened elsewhere across all live Profiles in the same browser endpoint.
287
- Extension-owned and browser-internal pages are excluded. Each JSON entry
288
- includes its opaque `profileContextId`; `bp tab <n>` selects any listed tab for
289
- the current Agent Lease. Its `selected` field is not a claim about Chrome's
290
- foreground tab or operating-system focus.
291
-
292
- ### Network
293
-
294
- | Command | Description |
295
- |---------|-------------|
296
- | `bp net` | List recent requests (`--url`, `--method`, `--status`, `--type`) |
297
- | `bp net show <id>` | Full request/response details (`--save <file>`) |
298
- | `bp net block <pattern>` | Block requests matching URL pattern |
299
- | `bp net mock <pattern>` | Mock responses (`--body`, `--file`) |
300
- | `bp net headers <pattern> <header...>` | Add/override request headers |
301
- | `bp net rules` | List active interception rules |
302
- | `bp net remove [id]` | Remove rule(s) (`--all`) |
303
- | `bp net clear` | Clear captured request log |
304
-
305
- ### Session
306
-
307
- | Command | Description |
308
- |---------|-------------|
309
- | `bp connect` | Connect to Chrome; create a Pilot window immediately only when Profile routing is unambiguous |
310
- | `bp disconnect` | Release this CLI namespace; stop the daemon only when no other client or active Lease is live |
311
- | `bp profiles` | Passively list live Chrome Profile contexts and representative tabs |
312
- | `bp profiles --identify [--refresh]` | Explicitly verify Profile/account identity using temporary visible Chrome pages |
313
- | `bp profile <index\|id\|label\|name\|email>` | Select a Profile for subsequent managed tabs |
314
- | `bp tabs` | List all controllable tabs in the current browser |
315
- | `bp tab <n>` | Select any listed managed or user tab for the current Agent Lease |
316
- | `bp close` | Close the current tab (`--all` closes Pilot-managed tabs only) |
317
-
318
- ## Refs
319
-
320
- Action commands return a snapshot of interactive elements, each with a `[ref]` number:
147
+ Or pass `--client-key` before the command. The key preserves the Agent's
148
+ selected profile, tab, frame, refs, auth, network rules, downloads, and command
149
+ history. Different keys are isolated and cannot overwrite one another.
321
150
 
322
- ```
323
- [1] link "Home"
324
- [2] textbox "Search" ← <input>, <textarea>, or contenteditable
325
- [3] textbox "" ← unnamed input (still interactive)
326
- [4] combobox "" ← <select> dropdown
327
- [5] spinbutton "Quantity" ← <input type="number">
328
- [6] button "Submit"
329
- [7] checkbox "Agree" checked
330
- [8] slider "Volume" ← <input type="range">
331
- ```
151
+ Browser Pilot keeps this state in background-service memory, not global
152
+ `state.json` or `refs.json` files. `bp disconnect` releases only the current
153
+ key. It stops the service only when no other Agent is using it and the invoking
154
+ executable owns that service process.
332
155
 
333
- Use the number in subsequent commands: `bp click 1`, `bp type 2 "hello"`.
156
+ Compatible global, local, and product-bundled CLIs reuse the same per-user
157
+ service through protocol negotiation. Executable identity is required only for
158
+ stopping the whole service. Use a distinct absolute `BROWSER_PILOT_HOME` only
159
+ for a deliberately incompatible isolated installation.
334
160
 
335
- Refs are scoped to the current page — they refresh automatically after every
336
- action. Elements inside Shadow DOM are included automatically. Snapshot JSON
337
- may also include a `page` block with viewport/document size, scroll position,
338
- remaining pixels, and scroll percentages.
161
+ ## Core Workflow
339
162
 
340
- Across one-shot CLI processes, selected target, frame, and refs live only in the
341
- daemon's keyed compatibility Workspace and renewable Lease. They are never
342
- written to `state.json` or `refs.json`; after Lease expiry the Agent must
343
- observe again. Set a distinct stable `BROWSER_PILOT_CLIENT_KEY` for each Agent
344
- that directly invokes the CLI. `bp disconnect` clears only that namespace and
345
- stops the Broker only when no other active Lease or embedded client remains.
163
+ ```bash
164
+ bp tabs # find existing user or managed tabs
165
+ bp tab 2 # select by current one-based index
166
+ bp open https://example.com --new # create independent managed work
167
+ bp snapshot # get controls with numbered refs
168
+ bp read main --limit 10000 # get bounded readable content
169
+ bp search "invoice total" # find text with nearby context
170
+ bp click 3 # act on a fresh ref
171
+ bp type 5 "hello" --clear # replace input and verify readback
172
+ bp press Enter # press a key and return fresh state
173
+ ```
346
174
 
347
- ## Output
175
+ Action commands return fresh page state and typed browser-visible evidence.
176
+ Refs belong to the current tab, frame, and page state. Refresh them after
177
+ navigation, document changes, tab/frame switches, reconnect, or `stale_ref`.
348
178
 
349
- **JSON by default** when piped (for LLM/script consumption). Human-readable when run in a terminal.
179
+ Use dedicated page primitives before JavaScript:
350
180
 
351
- ```json
352
- {"ok":true,"title":"Example","url":"https://example.com","page":{"viewportWidth":1280,"viewportHeight":720,"documentWidth":1280,"documentHeight":1800,"scrollX":0,"scrollY":0,"pixelsAbove":0,"pixelsBelow":1080,"pixelsLeft":0,"pixelsRight":0,"scrollPercentX":0,"scrollPercentY":0},"elements":[{"ref":1,"role":"link","name":"More info"}]}
181
+ ```bash
182
+ bp find ".result" --attributes href
183
+ bp scroll --to-text "Payment details"
184
+ bp dropdown 4
185
+ bp select 4 "United States"
186
+ bp screenshot page.png --annotate 1,3,8
353
187
  ```
354
188
 
355
- Machine errors include stable codes and may include hints or remediation:
356
- ```json
357
- {"ok":false,"error":"Ref [99] is stale.","code":"stale_ref","retryable":false,"context":{"field":"ref"},"hint":"Run 'bp snapshot' to refresh element refs."}
358
- ```
189
+ `bp eval` remains an escape hatch when no dedicated command represents the
190
+ operation.
359
191
 
360
- Branch on `code`, not English text. `retryable` means a later call may be valid;
361
- it does not make the same uncertain mutation safe to replay.
192
+ ## Async Work and Recovery
362
193
 
363
- Force human output: `bp --human open https://example.com`
194
+ Wait for browser-visible state instead of using shell sleeps:
364
195
 
365
- `bp screenshot [file]`, `bp pdf [file]`, and `bp net show <id> --save <file>`
366
- return an absolute local path. Screenshot and PDF bytes are first protected
367
- Artifacts, explicitly exported to that path, then released from Broker storage.
368
- `bp upload <file>` similarly imports a protected copy and releases it after the
369
- browser receives the file; the source file is never removed.
196
+ ```bash
197
+ bp --timeout 30000 wait --url "*/complete"
198
+ bp --timeout 30000 wait --text "Payment received"
199
+ bp --timeout 30000 wait --selector ".result"
200
+ bp --timeout 30000 wait --dialog
201
+ bp --timeout 120000 wait --download
202
+ bp --timeout 30000 wait --popup
203
+ ```
370
204
 
371
- After `bp snapshot`, `bp screenshot page.png --annotate` draws up to the first
372
- 200 current refs on a viewport image; pass a comma-separated list such as
373
- `--annotate 1,3,8` to limit it. Annotation does not inject elements into the
374
- page, runs in an isolated JavaScript world, and cannot be combined with full-page
375
- or selector capture.
205
+ Inspect service and command state after a timeout, interrupted shell call, or
206
+ lost result:
376
207
 
377
- ## Eval
208
+ ```bash
209
+ bp status
210
+ bp commands --status accepted,dispatched,unknown_outcome
211
+ bp command <command-id>
212
+ bp cancel <command-id>
213
+ ```
378
214
 
379
- `eval` is the escape hatch anything JavaScript can do:
215
+ Attach a stable request identity before the first attempt when the caller may
216
+ need to recover the same intended operation:
380
217
 
381
218
  ```bash
382
- bp eval "history.back()" # go back
383
- bp eval "history.forward()" # go forward
384
- bp eval "location.reload()" # reload
385
- bp eval "document.querySelector('h1').textContent" # extract text
386
- bp eval "document.querySelector('div').innerHTML" # extract HTML
387
- bp eval "JSON.stringify(localStorage)" # read storage
388
- echo 'complex js here' | bp eval # stdin for complex JS
219
+ bp --request-id task-42-submit click 7
389
220
  ```
390
221
 
391
- Prefer `bp read`, `bp search`, `bp find`, `bp scroll`, `bp dropdown`, and
392
- `bp select` for their bounded output, scoped state, and verification. Use
393
- `eval` only when no dedicated command represents the operation.
222
+ Reuse that ID only for recovery of the same intended call. Use a new ID for a
223
+ new action. Browser Pilot deduplicates stable requests and attempts
224
+ cancellation on CLI `SIGINT`/`SIGTERM`, but a dispatched mutation can still
225
+ have an `unknown_outcome`. Inspect the current page before retrying.
226
+
227
+ Machine errors always include stable `code` and `retryable` fields. Branch on
228
+ `code`, not English text.
394
229
 
395
- ## File Upload
230
+ ## Files, Screenshots, PDFs, and Downloads
396
231
 
397
- `bp upload` auto-detects `<input type="file">` on the page:
232
+ Use explicit task-owned paths or set a default directory:
398
233
 
399
234
  ```bash
400
- bp open https://images.google.com
401
- bp click 5 # click "Search by image"
402
- bp upload ~/Downloads/photo.jpg # auto-finds file input, triggers upload
235
+ export BROWSER_PILOT_OUTPUT_DIR=/absolute/task/output
236
+ bp screenshot
237
+ bp screenshot full.png --full
238
+ bp pdf report.pdf --landscape
239
+ bp downloads
240
+ bp download 1 invoice.pdf
241
+ bp net show 12 --save response.json
403
242
  ```
404
243
 
405
- ## Rich Text Editors & Shadow DOM
244
+ File results return an absolute `file` path, MIME type, byte size, and image
245
+ dimensions when available. An Agent can open that path with its normal image or
246
+ file-reading capability. CLI output remains bounded JSON and never embeds large
247
+ binary payloads.
406
248
 
407
- `bp type` works with contenteditable-based editors (Draft.js, ProseMirror, Quill, Slate, Lexical). They appear as `textbox` in snapshots:
249
+ Browser Pilot copies uploads and completed downloads into protected temporary
250
+ storage while processing them. It never removes the source upload or Chrome's
251
+ original download.
408
252
 
409
- ```bash
410
- bp type 3 "new content" --clear # replace content in a rich text editor
411
- ```
253
+ ## Command Groups
412
254
 
413
- For canvas-based editors (Google Docs, Google Sheets, Figma), use `bp keyboard` which sends real keyboard events:
255
+ | Area | Commands |
256
+ |------|----------|
257
+ | Setup and state | `browsers`, `connect`, `status`, `disconnect` |
258
+ | Profiles and tabs | `profiles`, `profile`, `tabs`, `tab`, `open`, `close` |
259
+ | Observation | `snapshot`, `read`, `search`, `find`, `locate` |
260
+ | Interaction | `click`, `type`, `keyboard`, `press`, `scroll`, `dropdown`, `select`, `eval` |
261
+ | Async and recovery | `wait`, `commands`, `command`, `cancel` |
262
+ | Page context | `frame`, `dialogs`, `dialog` |
263
+ | Files and media | `upload`, `downloads`, `download`, `screenshot`, `pdf` |
264
+ | Browser data | `cookies`, `auth`, `net` |
414
265
 
415
- ```bash
416
- bp keyboard "Hello Docs!" --click ".kix-appview-editor" # Google Docs
417
- bp press Meta+b # toggle bold
418
- bp keyboard "bold text"
419
- bp click --xy 400,300 # click canvas area
266
+ Run `bp --help` or `bp <command> --help` for exact options.
267
+
268
+ ## How It Works
269
+
270
+ ```text
271
+ short-lived bp CLI processes
272
+ |
273
+ | private local HTTP/Unix socket
274
+ v
275
+ per-user Browser Pilot service
276
+ |
277
+ | one supervised CDP WebSocket per browser endpoint
278
+ v
279
+ Chrome profiles and eligible tabs
420
280
  ```
421
281
 
422
- Shadow DOM elements are traversed automatically no special commands needed. Elements inside open shadow roots (even deeply nested) appear in snapshots and can be clicked/typed normally.
282
+ The service serializes actions per physical tab while allowing independent
283
+ tabs to progress concurrently. Each Agent receives isolated logical state and
284
+ opaque tab/ref addresses. The same physical user tab can be controlled by only
285
+ one Agent at a time.
423
286
 
424
- For `<select>` and exposed ARIA dropdowns, inspect and select with the dedicated
425
- commands:
287
+ An internal connection supervisor owns Browser Pilot-created tabs so it can
288
+ clean them up after a service crash without touching user-opened tabs. Browser
289
+ disconnect invalidates old tab, frame, and ref identities before reconnect.
426
290
 
427
- ```bash
428
- bp dropdown 4
429
- bp select 4 "Option 2"
430
- ```
291
+ See:
431
292
 
432
- ## Network Interception
293
+ - [Current architecture](docs/architecture/browser-pilot-platform-spec.md)
294
+ - [Universal Agent integration plan](docs/plans/universal-agent-integration.md)
295
+ - [Browser capability evolution](docs/plans/browser-capability-evolution.md)
296
+ - [Chrome profile routing](docs/plans/profile-context-routing.md)
433
297
 
434
- Monitor, block, and mock HTTP requests:
298
+ ## Comparison
435
299
 
436
- ```bash
437
- # Monitor traffic
438
- bp net # list recent requests
439
- bp net --url "*api*" --method POST # filter by URL and method
440
- bp net show 3 # full details + response body
441
-
442
- # Block requests
443
- bp net block "*tracking*" # block analytics/tracking
444
- bp net block "*ads*"
445
-
446
- # Mock API responses
447
- bp net mock "*api/data*" --body '{"ok":true}'
448
- bp net mock "*api/users*" --file mock.json
449
-
450
- # Override request headers
451
- bp net headers "*api*" "Authorization:Bearer test123"
452
-
453
- # Manage rules
454
- bp net rules # list active rules
455
- bp net remove <id-from-bp-net-rules> # rule IDs are opaque
456
- bp net remove --all # clear all rules
457
- bp net clear # clear captured request log
300
+ | | Browser Pilot | browser-use | Playwright MCP | Chrome DevTools MCP |
301
+ |---|---|---|---|---|
302
+ | Agent interface | Skill + CLI | Python library/Agent runtime | MCP tools | MCP tools |
303
+ | User's signed-in Chrome | Yes | Yes | Usually isolated browser | Depends on launch mode |
304
+ | Browser extension | No | No | No | No |
305
+ | Eligible user tabs | Yes | Browser-session dependent | Browser-session dependent | Yes |
306
+ | Progressive Agent guidance | Skill references | Library/Agent prompts | Tool descriptions | Tool descriptions |
307
+ | Action verification | Typed evidence + fresh state | State/history checks | Fresh snapshots | Manual inspection |
308
+ | Async recovery | Wait + command status + request IDs | Agent/controller lifecycle | Host/MCP lifecycle | Host/MCP lifecycle |
309
+ | Local media results | Paths + metadata | Library objects/paths | MCP image/resource results | MCP results |
310
+ | Network interception | Observe, block, mock, headers | Limited by controller | Supported | Supported |
311
+
312
+ Browser Pilot adopts several strong browser-use ideas without adopting its
313
+ Python Agent runtime: adaptive DOM/accessibility observation, fresh state after
314
+ actions, input readback, obstruction checks, stale-state invalidation,
315
+ watchdogs, and bounded outputs. The Agent interface remains the CLI.
316
+
317
+ ## Product Embedding
318
+
319
+ Bundle the native CLI or `browser-pilot-cli` plus a pinned Node runtime. Verify
320
+ the versioned release index and checksums. Put the bundled command directory on
321
+ the Agent's `PATH`, install the complete skill, and inject:
322
+
323
+ ```text
324
+ BROWSER_PILOT_CLIENT_KEY=<stable logical Agent identity>
325
+ BROWSER_PILOT_OUTPUT_DIR=<absolute task-owned directory>
458
326
  ```
459
327
 
460
- ## Testing
328
+ Do not map every CLI command into a native tool. The Agent already has a shell
329
+ tool, the skill controls context growth, and `bp --help` provides runtime
330
+ discovery. Agent-managed and product-bundled installations therefore exercise
331
+ the exact same implementation and guidance.
461
332
 
462
- Local release tests are deterministic. Real-site checks run separately as
463
- non-blocking canaries so a third-party outage cannot fail the release gate:
333
+ Each release ships a plugin archive, native archives, checksums, and a release
334
+ index binding the CLI version, compatible skill range, protocol range, and
335
+ supported assets. No `darwin-x64` artifact is produced.
336
+
337
+ ## Testing
464
338
 
465
339
  ```bash
466
- npm test # unit + local core, compat, and network gates
467
- npm run test:capabilities # isolated-Chrome quantitative capability gate
468
- npm run test:host-acceptance # two embedded hosts + CLI + Artifacts + reconnect
469
- npm run test:distribution # packed global/local/product-bundled launch modes
470
- npm run test:canary # real-site drift report; non-blocking
471
- npm run test:canary:strict # fail on drift or unavailability
472
- npm run test:integration # compatibility alias for strict canaries
473
- npm run test:all # release gates plus non-blocking canaries
340
+ npm run test:unit # TypeScript build and all Node tests
341
+ npm test # unit tests plus local Playwright gates
342
+ npm run test:capabilities # isolated-Chrome quantitative capability gate
343
+ npm run test:distribution # packed global/local/bundled CLI verification
344
+ npm run test:canary # non-blocking real-site drift report
345
+ npm run test:canary:strict
474
346
  ```
475
347
 
476
- Automated browser gates, including Playwright and stdio conformance, start their
477
- own headless Chrome with a temporary profile, random debugging port, and
478
- isolated Broker home. Teardown stops the Broker and Chrome and removes those
479
- files. Tests fail closed if that isolation is absent; they never attach to the
480
- user's Chrome by default. A maintainer can explicitly opt into a destructive
481
- manual compatibility run against the user's browser by setting
482
- `BROWSER_PILOT_TEST_USER_CHROME=1`.
348
+ Automated browser tests use isolated temporary Chrome profiles and Broker homes.
349
+ They never attach to the user's Chrome unless a maintainer explicitly opts into
350
+ a controlled manual test.
483
351
 
484
- Maintainers can build and verify a native artifact with an official,
485
- SEA-capable Node 22.17.0 runtime:
352
+ Build and verify a native artifact with an SEA-capable Node runtime:
486
353
 
487
354
  ```bash
488
355
  BROWSER_PILOT_SEA_NODE=/absolute/path/to/node npm run build:standalone
@@ -490,35 +357,21 @@ npm run verify:standalone
490
357
  npm run package:standalone
491
358
  ```
492
359
 
493
- For a new release, bump only the root package version through npm. The npm
494
- `version` lifecycle synchronizes every active distribution manifest from that
495
- source of truth:
360
+ The root `package.json` version is the release source of truth. The npm version
361
+ lifecycle synchronizes the lockfile, plugin manifests, skill compatibility
362
+ range, and marketplace entry:
496
363
 
497
364
  ```bash
498
365
  npm version "$BROWSER_PILOT_RELEASE_VERSION" --no-git-tag-version
499
366
  npm run release:check-version
500
- node scripts/verify-release-version.mjs
501
367
  ```
502
368
 
503
- The sync step updates the lockfile root, Agent plugin version and CLI peer
504
- range, Claude plugin/marketplace manifests, and skill `compatibility.json`.
505
- The current release becomes the skill's tested/minimum CLI version while the
506
- upper bound remains the next major version. Browser Pilot keeps public one-shot
507
- commands backward compatible within a major line, including additive `0.x`
508
- releases before 1.0. Historical release notes and version-specific fixtures are
509
- never rewritten.
510
-
511
- The canary report is written to
512
- `test-results/real-site-canary/report.json`. It distinguishes semantic drift,
513
- third-party unavailability, and runner errors. Set
514
- `BROWSER_PILOT_CANARY_REPORT` to use another report path.
515
-
516
369
  ## Requirements
517
370
 
518
- - Chrome 144+ / Edge / Brave (any Chromium-based browser)
519
- - Node.js >= 18
520
- - Remote debugging enabled (`chrome://inspect/#remote-debugging`)
521
- - Native archives: Apple Silicon macOS, x64 Linux, or x64 Windows; no Intel Mac
371
+ - Chrome 144+, Edge, Brave, or another supported Chromium browser
372
+ - Node.js 18 or newer for the npm distribution
373
+ - Remote debugging enabled at `chrome://inspect/#remote-debugging`
374
+ - Apple Silicon macOS, x64 Linux, or x64 Windows for native releases
522
375
 
523
376
  ## License
524
377