browser-pilot-cli 0.4.0 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,488 +1,363 @@
1
- # browser-pilot
1
+ # Browser Pilot
2
2
 
3
- Give your AI agent control of your real browser — with your logins, cookies, and extensions intact. No extension needed.
3
+ Give an AI Agent control of the user's real Chrome browser, including existing
4
+ logins, cookies, profiles, and extensions. No browser extension is required.
4
5
 
5
6
  ```bash
6
7
  npm install -g browser-pilot-cli@latest
7
8
  bp --version
8
9
  ```
9
10
 
10
- Each skill/plugin release declares a supported CLI range and the exact version
11
- it was tested with. Newer CLI releases remain usable within that range;
12
- embedded products should pin the tested version for reproducible builds.
13
- The native release targets Apple Silicon macOS, x64 Linux, and x64 Windows;
14
- Intel Mac is not supported.
11
+ Native releases support Apple Silicon macOS, x64 Linux, and x64 Windows. Intel
12
+ Mac is not supported.
15
13
 
16
14
  ## Agent Setup
17
15
 
18
- ### 1. Enable Chrome remote debugging (one-time)
16
+ ### 1. Enable Chrome remote debugging
19
17
 
20
- Open `chrome://inspect/#remote-debugging` in Chrome (144+) and enable remote
21
- debugging. No command-line flags or restart are needed. Chrome may show its
22
- separate Allow dialog when `bp connect` requests the actual connection.
18
+ Open `chrome://inspect/#remote-debugging` in Chrome 144 or newer and enable
19
+ remote debugging. Browser Pilot does not use command-line debugging flags or a
20
+ Chrome extension.
23
21
 
24
- > Chrome 136 disabled the old `--remote-debugging-port` flag for security. Chrome 144 introduced this new UI toggle as the replacement — browser-pilot uses this.
25
-
26
- Check every installed supported browser and its exact setup state with:
22
+ Inspect installed browsers without requesting authorization:
27
23
 
28
24
  ```bash
29
25
  bp browsers
30
26
  ```
31
27
 
32
- Discovery is passive: `bp browsers`, bridge startup, and `initialize` never open
33
- a Chrome WebSocket and never display the Allow dialog.
34
-
35
- Use `bp connect --browser <id|product|channel>` for an explicit choice. Product
36
- embedders can pass the same selector to `browser-pilot bridge --stdio
37
- --browser <selector>`; otherwise selection follows a stable platform order.
38
- One connection covers every live Chrome Profile on that endpoint. When several
39
- Profiles are open, `bp connect` lists them without creating a Pilot window. Use
40
- the passive `bp profiles` inventory first. Run `bp profiles --identify` only
41
- when account-aware names are needed; it briefly opens and closes one visible
42
- `chrome://version` page per unidentified Profile. Profile and tab indexes are
43
- one-based. Select with `bp profile <index>`, or pass `--profile` to
44
- `bp open --new`.
28
+ Follow the returned remediation first. Start the selected browser or enable
29
+ remote debugging when requested. Browser Pilot does not connect automatically.
45
30
 
46
- ### 2. Install the plugin for your agent
31
+ When the browser is ready and browser work is needed, tell the user to watch
32
+ for Chrome's Allow dialog, then connect once:
47
33
 
48
- **Claude Code:**
34
+ ```bash
35
+ bp connect --browser <browser-id>
49
36
  ```
37
+
38
+ Chrome may display one Allow dialog for the browser connection. Browser
39
+ discovery and background-service startup are passive. Concurrent clients share
40
+ one pending connection request, so they do not create an authorization loop.
41
+ Machine-readable `bp connect` output is emitted after the attempt finishes, so
42
+ an Agent must show the Allow reminder before invoking the command. A timeout or
43
+ `browser_not_authorized` result must be inspected; it is not a reason to loop
44
+ connection attempts.
45
+
46
+ ### 2. Install the Agent skill
47
+
48
+ Claude Code:
49
+
50
+ ```text
50
51
  /plugin marketplace add https://github.com/relixiaobo/browser-pilot.git
51
52
  /plugin install browser-pilot@browser-pilot-marketplace
52
53
  ```
53
54
 
54
- **Codex CLI:**
55
+ Codex, Cursor, or VS Code Copilot:
56
+
55
57
  ```bash
56
58
  npx skills add relixiaobo/browser-pilot --skill browser-pilot
57
59
  ```
58
60
 
59
- **OpenClaw:**
61
+ OpenClaw or another Agent Skills-compatible host:
62
+
60
63
  ```bash
61
64
  cp -r plugin/skills/browser-pilot ~/.agents/skills/
62
65
  ```
63
66
 
64
- **Cursor / VS Code Copilot:**
65
- ```bash
66
- npx skills add relixiaobo/browser-pilot --skill browser-pilot
67
- ```
67
+ ### 3. Ask for the browser task
68
68
 
69
- ### 3. Use it
69
+ - "Fill in the form I already opened."
70
+ - "Check my GitHub notifications."
71
+ - "Download the latest invoice and summarize it."
72
+ - "Open this report in a new tab and save it as PDF."
70
73
 
71
- Just tell your agent what you want to do:
74
+ The Agent uses its existing shell tool to run `bp`. Browser Pilot does not add
75
+ native Agent tools, an MCP server, an SDK, or a persistent client protocol.
72
76
 
73
- - "Open GitHub and check my notifications"
74
- - "Go to Hacker News and summarize the top 5 posts"
75
- - "Fill out the form on this page"
77
+ ## Public Interface
76
78
 
77
- The agent will use `bp` commands automatically. Your real login sessions are preserved — no need to re-authenticate.
79
+ Browser Pilot has one Agent integration surface:
78
80
 
79
- ## Why browser-pilot?
81
+ ```text
82
+ Agent -> Browser Pilot skill -> existing shell/command runner
83
+ -> bp CLI -> shared background service -> Chrome
84
+ ```
80
85
 
81
- - **No extension required** Uses Chrome 144's native remote debugging toggle, not the Extension Debugger API
82
- - **Real login sessions** Operates your actual browser profile. Cookies, extensions, logins all intact
83
- - **Multi-Profile routing** Lists user tabs across live Chrome Profiles, explicitly verifies account-aware Profile identity, and creates managed work only in a resolved Profile
84
- - **CLI-native** Any agent with bash access can use it. No MCP protocol, no SDK integration needed
85
- - **Auto-snapshot** — Every action returns page state with numbered `[ref]` elements, so the agent always knows what's on screen
86
- - **Adaptive page views** — Bounded read, text search, DOM metadata, page geometry, and annotated screenshots let an agent choose the smallest useful representation
87
- - **Verified page primitives** — Scroll and dropdown tools return fresh state and typed evidence instead of requiring ad hoc JavaScript
88
- - **Lightweight npm install** — About 200KB as an npm tarball. No bundled Chromium (unlike Playwright's 400MB+)
89
- - **Rich editor support** — Works with contenteditable editors (Draft.js, ProseMirror, Quill, Slate) and Shadow DOM elements out of the box
86
+ The skill provides progressive disclosure and browser-specific operating
87
+ guidance. The CLI provides stable JSON, errors, state, files, waiting, and
88
+ recovery. The background service is private implementation detail shared by all
89
+ compatible CLI installations for the current OS user.
90
90
 
91
- ## Comparison
91
+ This same path supports both installation modes:
92
92
 
93
- | | browser-pilot | Playwright MCP | Chrome DevTools MCP | browser-use |
94
- |---|---|---|---|---|
95
- | **Interface** | CLI (bash) | MCP protocol | MCP protocol | Python SDK |
96
- | **Login session reuse** | Yes | No | Depends | Yes |
97
- | **Extension required** | No | No | No | No |
98
- | **Element refs** | Numbered (accessibility tree) | Named refs (ARIA) | CSS selectors | Numbered (DOM) |
99
- | **Auto-snapshot after action** | Yes | Yes | No | Yes |
100
- | **Network interception** | Yes (block/mock/headers) | Yes | Yes | No |
101
- | **Multi-browser** | Chromium-only | Chromium + Firefox + WebKit | Chromium-only | Chromium-only |
102
- | **Dialog handling** | Explicit | Automatic | Manual | Automatic |
103
- | **JSON output** | Default | MCP structured | MCP structured | Python objects |
104
- | **File upload** | Auto-detect input | Yes | No | Yes |
93
+ - An Agent installs `browser-pilot-cli` itself.
94
+ - A product bundles a pinned CLI and places its directory on the Agent command
95
+ environment's `PATH`.
105
96
 
106
- ## How It Works
97
+ Tenon, OpenClaw, Codex, and other shell-capable Agents need no Browser
98
+ Pilot-specific runtime integration. See
99
+ [the embedding reference](plugin/skills/browser-pilot/references/embedding.md).
107
100
 
108
- ```
109
- Agent (bash tool)
110
- │ bp open / bp click / bp eval ...
111
-
112
- CLI Process ──── HTTP/Unix Socket ──── Daemon Process (persistent)
113
-
114
- │ WebSocket (CDP)
115
-
116
- Chrome (your browser, your Profiles)
117
- ├── Eligible user tabs (selectable)
118
- └── Managed windows (Broker-owned)
119
- ```
101
+ ## Browser Scope
120
102
 
121
- The Broker maintains isolated state for each Agent. A private connection
122
- supervisor owns the Broker's single browser-level CDP WebSocket and proxies it
123
- to the daemon. It also owns only Browser Pilot-created tabs so it can clean them
124
- after a Broker crash without touching user-opened tabs. Controller identity is
125
- returned to the embedding Host; Browser Pilot never injects a status border or
126
- other control UI into the page DOM.
103
+ `bp tabs` exposes all eligible page tabs on the authorized browser endpoint:
127
104
 
128
- ## Platform Evolution
105
+ - Browser Pilot-created managed tabs;
106
+ - eligible popups from managed tabs;
107
+ - eligible tabs the user opened in any live Chrome profile.
129
108
 
130
- Browser Pilot now routes direct `bp` commands and embedded clients through the
131
- same Agent-neutral, multi-client Browser Broker.
132
- The approved architecture and execution plans are:
109
+ The Agent can therefore fill a form or inspect a page the user already opened.
110
+ Browser-internal and extension-owned pages are excluded. Existing Chrome tab
111
+ groups do not limit tab control, whether expanded or collapsed, but Browser
112
+ Pilot cannot inspect or manage tab groups without an extension.
133
113
 
134
- - [Platform specification](docs/architecture/browser-pilot-platform-spec.md)
135
- - [Universal Agent integration plan](docs/plans/universal-agent-integration.md)
136
- - [Browser capability and reliability plan](docs/plans/browser-capability-evolution.md)
137
- - [Stdio bridge integration contract](docs/integration/stdio-bridge.md)
138
- - [Stdio black-box conformance suite](docs/integration/stdio-conformance.md)
139
- - [Reference consumer adapters](https://github.com/relixiaobo/browser-pilot/tree/main/examples/adapters),
140
- including Tenon and OpenClaw lifecycle mappings
141
-
142
- The public integration direction remains CLI-only: one-shot commands for direct
143
- Agent use and a persistent `bridge --stdio` mode for products that embed the
144
- official executable. Browser Pilot will not require an extension, Native SDK,
145
- or MCP server. The repository's consumer adapters are source examples built
146
- only on the stdio contract; they are intentionally excluded from the npm
147
- package and do not create a second public library API.
148
-
149
- The dedicated Pilot window remains the default managed tab set for independent
150
- Agent work. The Broker architecture also includes all eligible user tabs in the
151
- Agent's inventory without a separate grant step, so an Agent can operate a page
152
- the user already opened. Invoking or exposing the tool is the authorization
153
- boundary; products may apply their own approval UX or remove operations when
154
- launching the bridge. Bulk cleanup remains limited to managed tabs, and user
155
- tabs remain open when a session ends.
156
-
157
- The `browser-pilot bridge --stdio` transport, Broker lifecycle, browser tool
158
- dispatch, event replay, protected Artifacts, protocol 1.1 transport limit
159
- negotiation, protocol 1.2 Chrome Profile routing, and protocol 1.3 verified
160
- Profile identity and unambiguous result vocabulary are implemented. Browser
161
- disconnect and explicit reconnect handling, scoped
162
- download Artifacts, Workspace resource isolation, and typed watchdog events for
163
- stalled navigation, selected-frame detach, pending dialogs, and repeated
164
- browser-observable no-progress actions are also implemented. Bounded page
165
- search, DOM metadata lookup, scrolling, native/ARIA dropdown operations,
166
- Observation page geometry, and viewport screenshot annotations adapt the most
167
- useful browser-use inspection patterns to the same scoped Broker contract. An
168
- internal managed-target janitor is the sole owner of the browser-level CDP
169
- connection, proxies daemon traffic over private IPC, and closes only
170
- Broker-created tabs and their managed popup descendants if the daemon exits or
171
- crashes. It does not persist target IDs or close user tabs. Broker startup and
172
- browser discovery are passive. Only `bp connect` or the versioned
173
- `browser.connect` tool requests Chrome authorization; concurrent clients share
174
- one in-flight request, and a failed or dropped connection is never retried by a
175
- timer.
176
- Compatible Agent products reuse one per-user Broker through protocol
177
- negotiation. Protected shutdown cannot terminate live embedded clients or
178
- another CLI Agent's active Lease, and an
179
- incompatible product must explicitly select a separate `BROWSER_PILOT_HOME`.
180
- Global npm installation, local npm/npx use, and product-bundled absolute-path
181
- launches are covered by distribution black-box tests. Native self-contained
182
- release artifacts use one executable for the public CLI and its private Broker
183
- and janitor roles, with no runtime download or system-Node fallback.
184
-
185
- ## Installation and Embedding
186
-
187
- Agent-managed installation remains supported:
114
+ Browser Pilot does not implement task-intent permission prompts. Invoking the
115
+ tool gives the Agent the full eligible browser capability surface. An Agent
116
+ host may apply its own policy or approval UX, but that is outside the Browser
117
+ Pilot contract.
188
118
 
189
- ```bash
190
- npm install -g browser-pilot-cli@latest
191
- bp --version
192
- bp connect
193
- bp tabs
194
- ```
119
+ User-opened tabs remain open when Agent state is released. `bp close --all`
120
+ closes only Browser Pilot-created tabs. `bp close` explicitly closes the
121
+ selected tab, including a user tab.
122
+
123
+ ## Chrome Profiles
195
124
 
196
- A project can pin Browser Pilot locally and run the same CLI without a global
197
- installation:
125
+ One authorized browser connection can expose several live Chrome profiles.
198
126
 
199
127
  ```bash
200
- npm install --save-exact browser-pilot-cli@latest
201
- npx --no-install browser-pilot tabs
128
+ bp profiles
129
+ bp profiles --identify
130
+ bp profile 2
131
+ bp open https://example.com --new --profile 2
202
132
  ```
203
133
 
204
- Agent products should pin either the npm package or a native archive from the
205
- GitHub release. Launch an absolute path directly with `bridge --stdio`; do not
206
- invoke a shell, import `src/*`, or depend on Broker locator/socket files. The
207
- npm form launches `node_modules/browser-pilot-cli/dist/cli.js` with the
208
- product's pinned Node runtime. The native archive launches its single
209
- `browser-pilot` executable and does not require Node to be installed.
210
-
211
- Each native archive contains `manifest.json`, per-file `SHA256SUMS`, licenses,
212
- and an adjacent archive checksum. The manifest reports the actual signature
213
- kind: Developer ID or ad-hoc on macOS, Authenticode or unsigned on Windows, and
214
- unsigned on Linux. Release automation never labels an artifact signed when its
215
- signing credentials were unavailable.
216
-
217
- The same GitHub release also contains
218
- `browser-pilot-plugin-<version>.tgz`, its checksum, and a versioned release index
219
- that binds the npm version, protocol range, compatible skill/plugin range,
220
- tested CLI version, and the three supported native assets. Consumers should
221
- verify that index before bundling. No `darwin-x64` asset is produced.
222
-
223
- All three forms use the same protocol and compatible per-user Broker. Use a
224
- distinct absolute `BROWSER_PILOT_HOME` only when a product deliberately needs
225
- an incompatible isolated Broker.
226
-
227
- Products embedding `bridge --stdio` already receive isolated Connections,
228
- Workspaces, and Leases. When several independent Agents instead invoke the
229
- one-shot CLI directly, assign each integration a distinct stable namespace:
134
+ `bp profiles` is passive and includes representative tabs. Use
135
+ `--identify` only when account-aware identity is needed; it briefly opens and
136
+ closes a visible `chrome://version` page per unidentified profile and accepts
137
+ identity only after exact profile-path verification. Browser Pilot reports
138
+ verified profile/account names rather than forcing the Agent to speak in
139
+ ambiguous "Profile 1" labels.
140
+
141
+ Profile selection controls where new managed tabs are created. It does not
142
+ grant or restrict access to existing eligible tabs.
143
+
144
+ ## Multi-Agent State
145
+
146
+ Every independent Agent must use one stable key across all short-lived CLI
147
+ processes:
230
148
 
231
149
  ```bash
232
- BROWSER_PILOT_CLIENT_KEY=agent.product.install-01 bp tabs
233
- bp --client-key agent.product.install-01 snapshot
150
+ export BROWSER_PILOT_CLIENT_KEY=product.install.agent-01
151
+ bp tabs
152
+ bp snapshot
234
153
  ```
235
154
 
236
- Repeated commands with the same key reuse that Agent's selected target, frame,
237
- refs, auth, and network state. Different keys cannot overwrite one another.
238
- The default key remains `browser-pilot-cli` for a single interactive Agent.
239
-
240
- ## Commands
241
-
242
- ### Core Loop
243
-
244
- | Command | Returns | Description |
245
- |---------|---------|-------------|
246
- | `bp open <url>` | snapshot | Navigate to URL |
247
- | `bp open <url> --new --profile <selector>` | snapshot | Create managed work in one live Chrome Profile |
248
- | `bp snapshot` | snapshot | Get interactive elements |
249
- | `bp read [selector]` | text | Get bounded readable page/region content |
250
- | `bp search <text>` | matches | Find bounded visible text and nearby context |
251
- | `bp find <selector>` | elements | Inspect bounded DOM metadata and requested attributes |
252
- | `bp scroll [direction]` | snapshot | Scroll page/element/text and return fresh state |
253
- | `bp dropdown <target>` | options | List native or exposed ARIA dropdown options |
254
- | `bp select <target> <option>` | snapshot | Select and verify a dropdown option |
255
- | `bp click <ref>` | snapshot | Click element by ref number (`--double`, `--right`) |
256
- | `bp click --xy x,y` | snapshot | Click at viewport coordinates (canvas, maps) |
257
- | `bp locate <selector>` | coords | Get element center x,y + size (for `click --xy`) |
258
- | `bp type <ref> <text>` | snapshot | Type into element (`--clear`, `--submit`) |
259
- | `bp keyboard <text>` | snapshot | Type via keyboard events (`--click`, `--clear`) |
260
- | `bp press <key>` | snapshot | Press key (Enter, Escape, Control+a, Meta+c) |
261
- | `bp eval [js]` | value | Run JavaScript (escape hatch for anything) |
262
-
263
- ### Utilities
264
-
265
- | Command | Description |
266
- |---------|-------------|
267
- | `bp screenshot [file]` | Capture screenshot (`--full`, `--selector`, `--annotate [refs]`) |
268
- | `bp pdf [file]` | Save page as PDF (`--landscape`) |
269
- | `bp cookies [domain]` | View cookies (includes HttpOnly) |
270
-
271
- ### Edge Cases
272
-
273
- | Command | Description |
274
- |---------|-------------|
275
- | `bp upload <filepath>` | Upload file (auto-finds `<input type="file">`) |
276
- | `bp auth <user> <pass>` | Set HTTP Basic Auth credentials (`--clear`) |
277
- | `bp frame [index]` | List or switch iframe context (0 = top) |
278
- | `bp dialogs` | List pending JavaScript dialogs |
279
- | `bp dialog <id> --accept\|--dismiss` | Explicitly respond to a dialog (`--prompt`) |
280
-
281
- Dialogs remain pending until explicitly accepted or dismissed. One-shot CLI
282
- dialogs are scoped to the compatibility Workspace and isolated from embedded
283
- Broker clients.
284
-
285
- Run `bp tabs` to list Pilot-managed tabs, their popups, and eligible tabs the
286
- user opened elsewhere across all live Profiles in the same browser endpoint.
287
- Extension-owned and browser-internal pages are excluded. Each JSON entry
288
- includes its opaque `profileContextId`; `bp tab <n>` selects any listed tab for
289
- the current Agent Lease. Its `selected` field is not a claim about Chrome's
290
- foreground tab or operating-system focus.
291
-
292
- ### Network
293
-
294
- | Command | Description |
295
- |---------|-------------|
296
- | `bp net` | List recent requests (`--url`, `--method`, `--status`, `--type`) |
297
- | `bp net show <id>` | Full request/response details (`--save <file>`) |
298
- | `bp net block <pattern>` | Block requests matching URL pattern |
299
- | `bp net mock <pattern>` | Mock responses (`--body`, `--file`) |
300
- | `bp net headers <pattern> <header...>` | Add/override request headers |
301
- | `bp net rules` | List active interception rules |
302
- | `bp net remove [id]` | Remove rule(s) (`--all`) |
303
- | `bp net clear` | Clear captured request log |
304
-
305
- ### Session
306
-
307
- | Command | Description |
308
- |---------|-------------|
309
- | `bp connect` | Connect to Chrome; create a Pilot window immediately only when Profile routing is unambiguous |
310
- | `bp disconnect` | Release this CLI namespace; stop the daemon only when no other client or active Lease is live |
311
- | `bp profiles` | Passively list live Chrome Profile contexts and representative tabs |
312
- | `bp profiles --identify [--refresh]` | Explicitly verify Profile/account identity using temporary visible Chrome pages |
313
- | `bp profile <index\|id\|label\|name\|email>` | Select a Profile for subsequent managed tabs |
314
- | `bp tabs` | List all controllable tabs in the current browser |
315
- | `bp tab <n>` | Select any listed managed or user tab for the current Agent Lease |
316
- | `bp close` | Close the current tab (`--all` closes Pilot-managed tabs only) |
317
-
318
- ## Refs
319
-
320
- Action commands return a snapshot of interactive elements, each with a `[ref]` number:
155
+ Or pass `--client-key` before the command. The key preserves the Agent's
156
+ selected profile, tab, frame, refs, auth, network rules, downloads, and command
157
+ history. Different keys are isolated and cannot overwrite one another.
321
158
 
322
- ```
323
- [1] link "Home"
324
- [2] textbox "Search" ← <input>, <textarea>, or contenteditable
325
- [3] textbox "" ← unnamed input (still interactive)
326
- [4] combobox "" ← <select> dropdown
327
- [5] spinbutton "Quantity" ← <input type="number">
328
- [6] button "Submit"
329
- [7] checkbox "Agree" checked
330
- [8] slider "Volume" ← <input type="range">
331
- ```
159
+ Browser Pilot keeps this state in background-service memory, not global
160
+ `state.json` or `refs.json` files. `bp disconnect` releases only the current
161
+ key. It stops the service only when no other Agent is using it and the invoking
162
+ executable owns that service process.
332
163
 
333
- Use the number in subsequent commands: `bp click 1`, `bp type 2 "hello"`.
164
+ Compatible global, local, and product-bundled CLIs reuse the same per-user
165
+ service through protocol negotiation. Executable identity is required only for
166
+ stopping the whole service. Use a distinct absolute `BROWSER_PILOT_HOME` only
167
+ for a deliberately incompatible isolated installation.
334
168
 
335
- Refs are scoped to the current page — they refresh automatically after every
336
- action. Elements inside Shadow DOM are included automatically. Snapshot JSON
337
- may also include a `page` block with viewport/document size, scroll position,
338
- remaining pixels, and scroll percentages.
169
+ ## Core Workflow
339
170
 
340
- Across one-shot CLI processes, selected target, frame, and refs live only in the
341
- daemon's keyed compatibility Workspace and renewable Lease. They are never
342
- written to `state.json` or `refs.json`; after Lease expiry the Agent must
343
- observe again. Set a distinct stable `BROWSER_PILOT_CLIENT_KEY` for each Agent
344
- that directly invokes the CLI. `bp disconnect` clears only that namespace and
345
- stops the Broker only when no other active Lease or embedded client remains.
171
+ ```bash
172
+ bp tabs # find existing user or managed tabs
173
+ bp tab 2 # select by current one-based index
174
+ bp open https://example.com --new # create independent managed work
175
+ bp snapshot # get controls with numbered refs
176
+ bp read main --limit 10000 # get bounded readable content
177
+ bp search "invoice total" # find text with nearby context
178
+ bp click 3 # act on a fresh ref
179
+ bp type 5 "hello" --clear # replace input and verify readback
180
+ bp press Enter # press a key and return fresh state
181
+ ```
346
182
 
347
- ## Output
183
+ Action commands return fresh page state and typed browser-visible evidence.
184
+ Refs belong to the current tab, frame, and page state. Refresh them after
185
+ navigation, document changes, tab/frame switches, reconnect, or `stale_ref`.
348
186
 
349
- **JSON by default** when piped (for LLM/script consumption). Human-readable when run in a terminal.
187
+ Use dedicated page primitives before JavaScript:
350
188
 
351
- ```json
352
- {"ok":true,"title":"Example","url":"https://example.com","page":{"viewportWidth":1280,"viewportHeight":720,"documentWidth":1280,"documentHeight":1800,"scrollX":0,"scrollY":0,"pixelsAbove":0,"pixelsBelow":1080,"pixelsLeft":0,"pixelsRight":0,"scrollPercentX":0,"scrollPercentY":0},"elements":[{"ref":1,"role":"link","name":"More info"}]}
189
+ ```bash
190
+ bp find ".result" --attributes href
191
+ bp scroll --to-text "Payment details"
192
+ bp dropdown 4
193
+ bp select 4 "United States"
194
+ bp screenshot page.png --annotate 1,3,8
353
195
  ```
354
196
 
355
- Machine errors include stable codes and may include hints or remediation:
356
- ```json
357
- {"ok":false,"error":"Ref [99] is stale.","code":"stale_ref","retryable":false,"context":{"field":"ref"},"hint":"Run 'bp snapshot' to refresh element refs."}
358
- ```
197
+ `bp eval` remains an escape hatch when no dedicated command represents the
198
+ operation.
359
199
 
360
- Branch on `code`, not English text. `retryable` means a later call may be valid;
361
- it does not make the same uncertain mutation safe to replay.
200
+ ## Async Work and Recovery
362
201
 
363
- Force human output: `bp --human open https://example.com`
202
+ Wait for browser-visible state instead of using shell sleeps:
364
203
 
365
- `bp screenshot [file]`, `bp pdf [file]`, and `bp net show <id> --save <file>`
366
- return an absolute local path. Screenshot and PDF bytes are first protected
367
- Artifacts, explicitly exported to that path, then released from Broker storage.
368
- `bp upload <file>` similarly imports a protected copy and releases it after the
369
- browser receives the file; the source file is never removed.
204
+ ```bash
205
+ bp --timeout 30000 wait --url "*/complete"
206
+ bp --timeout 30000 wait --text "Payment received"
207
+ bp --timeout 30000 wait --selector ".result"
208
+ bp --timeout 30000 wait --dialog
209
+ bp --timeout 120000 wait --download
210
+ bp --timeout 30000 wait --popup
211
+ ```
370
212
 
371
- After `bp snapshot`, `bp screenshot page.png --annotate` draws up to the first
372
- 200 current refs on a viewport image; pass a comma-separated list such as
373
- `--annotate 1,3,8` to limit it. Annotation does not inject elements into the
374
- page, runs in an isolated JavaScript world, and cannot be combined with full-page
375
- or selector capture.
213
+ Inspect service and command state after a timeout, interrupted shell call, or
214
+ lost result:
376
215
 
377
- ## Eval
216
+ ```bash
217
+ bp status
218
+ bp commands --status accepted,dispatched,unknown_outcome
219
+ bp command <command-id>
220
+ bp cancel <command-id>
221
+ ```
378
222
 
379
- `eval` is the escape hatch anything JavaScript can do:
223
+ Attach a stable request identity before the first attempt when the caller may
224
+ need to recover the same intended operation:
380
225
 
381
226
  ```bash
382
- bp eval "history.back()" # go back
383
- bp eval "history.forward()" # go forward
384
- bp eval "location.reload()" # reload
385
- bp eval "document.querySelector('h1').textContent" # extract text
386
- bp eval "document.querySelector('div').innerHTML" # extract HTML
387
- bp eval "JSON.stringify(localStorage)" # read storage
388
- echo 'complex js here' | bp eval # stdin for complex JS
227
+ bp --request-id task-42-submit click 7
389
228
  ```
390
229
 
391
- Prefer `bp read`, `bp search`, `bp find`, `bp scroll`, `bp dropdown`, and
392
- `bp select` for their bounded output, scoped state, and verification. Use
393
- `eval` only when no dedicated command represents the operation.
230
+ Reuse that ID only for recovery of the same intended call. Use a new ID for a
231
+ new action. Browser Pilot deduplicates stable requests and attempts
232
+ cancellation on CLI `SIGINT`/`SIGTERM`, but a dispatched mutation can still
233
+ have an `unknown_outcome`. Inspect the current page before retrying.
394
234
 
395
- ## File Upload
235
+ Machine errors always include stable `code` and `retryable` fields. Branch on
236
+ `code`, not English text.
396
237
 
397
- `bp upload` auto-detects `<input type="file">` on the page:
238
+ ## Files, Screenshots, PDFs, and Downloads
239
+
240
+ Use explicit task-owned paths or set a default directory:
398
241
 
399
242
  ```bash
400
- bp open https://images.google.com
401
- bp click 5 # click "Search by image"
402
- bp upload ~/Downloads/photo.jpg # auto-finds file input, triggers upload
243
+ export BROWSER_PILOT_OUTPUT_DIR=/absolute/task/output
244
+ bp screenshot
245
+ bp screenshot full.png --full
246
+ bp pdf report.pdf --landscape
247
+ bp downloads
248
+ bp download 1 invoice.pdf
249
+ bp net show 12 --save response.json
403
250
  ```
404
251
 
405
- ## Rich Text Editors & Shadow DOM
252
+ File results return an absolute `file` path, MIME type, byte size, and image
253
+ dimensions when available. An Agent can open that path with its normal image or
254
+ file-reading capability. CLI output remains bounded JSON and never embeds large
255
+ binary payloads.
406
256
 
407
- `bp type` works with contenteditable-based editors (Draft.js, ProseMirror, Quill, Slate, Lexical). They appear as `textbox` in snapshots:
257
+ Browser Pilot copies uploads and completed downloads into protected temporary
258
+ storage while processing them. It never removes the source upload or Chrome's
259
+ original download.
408
260
 
409
- ```bash
410
- bp type 3 "new content" --clear # replace content in a rich text editor
411
- ```
261
+ ## Command Groups
412
262
 
413
- For canvas-based editors (Google Docs, Google Sheets, Figma), use `bp keyboard` which sends real keyboard events:
263
+ | Area | Commands |
264
+ |------|----------|
265
+ | Setup and state | `browsers`, `connect`, `status`, `disconnect` |
266
+ | Profiles and tabs | `profiles`, `profile`, `tabs`, `tab`, `open`, `close` |
267
+ | Observation | `snapshot`, `read`, `search`, `find`, `locate` |
268
+ | Interaction | `click`, `type`, `keyboard`, `press`, `scroll`, `dropdown`, `select`, `eval` |
269
+ | Async and recovery | `wait`, `commands`, `command`, `cancel` |
270
+ | Page context | `frame`, `dialogs`, `dialog` |
271
+ | Files and media | `upload`, `downloads`, `download`, `screenshot`, `pdf` |
272
+ | Browser data | `cookies`, `auth`, `net` |
414
273
 
415
- ```bash
416
- bp keyboard "Hello Docs!" --click ".kix-appview-editor" # Google Docs
417
- bp press Meta+b # toggle bold
418
- bp keyboard "bold text"
419
- bp click --xy 400,300 # click canvas area
274
+ Run `bp --help` or `bp <command> --help` for exact options.
275
+
276
+ ## How It Works
277
+
278
+ ```text
279
+ short-lived bp CLI processes
280
+ |
281
+ | private local HTTP/Unix socket
282
+ v
283
+ per-user Browser Pilot service
284
+ |
285
+ | one supervised CDP WebSocket per browser endpoint
286
+ v
287
+ Chrome profiles and eligible tabs
420
288
  ```
421
289
 
422
- Shadow DOM elements are traversed automatically no special commands needed. Elements inside open shadow roots (even deeply nested) appear in snapshots and can be clicked/typed normally.
290
+ The service serializes actions per physical tab while allowing independent
291
+ tabs to progress concurrently. Each Agent receives isolated logical state and
292
+ opaque tab/ref addresses. The same physical user tab can be controlled by only
293
+ one Agent at a time.
423
294
 
424
- For `<select>` and exposed ARIA dropdowns, inspect and select with the dedicated
425
- commands:
295
+ An internal connection supervisor owns Browser Pilot-created tabs so it can
296
+ clean them up after a service crash without touching user-opened tabs. Browser
297
+ disconnect invalidates old tab, frame, and ref identities before reconnect.
426
298
 
427
- ```bash
428
- bp dropdown 4
429
- bp select 4 "Option 2"
430
- ```
299
+ See:
431
300
 
432
- ## Network Interception
301
+ - [Current architecture](docs/architecture/browser-pilot-platform-spec.md)
302
+ - [Universal Agent integration plan](docs/plans/universal-agent-integration.md)
303
+ - [Browser capability evolution](docs/plans/browser-capability-evolution.md)
304
+ - [Chrome profile routing](docs/plans/profile-context-routing.md)
433
305
 
434
- Monitor, block, and mock HTTP requests:
306
+ ## Comparison
435
307
 
436
- ```bash
437
- # Monitor traffic
438
- bp net # list recent requests
439
- bp net --url "*api*" --method POST # filter by URL and method
440
- bp net show 3 # full details + response body
441
-
442
- # Block requests
443
- bp net block "*tracking*" # block analytics/tracking
444
- bp net block "*ads*"
445
-
446
- # Mock API responses
447
- bp net mock "*api/data*" --body '{"ok":true}'
448
- bp net mock "*api/users*" --file mock.json
449
-
450
- # Override request headers
451
- bp net headers "*api*" "Authorization:Bearer test123"
452
-
453
- # Manage rules
454
- bp net rules # list active rules
455
- bp net remove <id-from-bp-net-rules> # rule IDs are opaque
456
- bp net remove --all # clear all rules
457
- bp net clear # clear captured request log
308
+ | | Browser Pilot | browser-use | Playwright MCP | Chrome DevTools MCP |
309
+ |---|---|---|---|---|
310
+ | Agent interface | Skill + CLI | Python library/Agent runtime | MCP tools | MCP tools |
311
+ | User's signed-in Chrome | Yes | Yes | Usually isolated browser | Depends on launch mode |
312
+ | Browser extension | No | No | No | No |
313
+ | Eligible user tabs | Yes | Browser-session dependent | Browser-session dependent | Yes |
314
+ | Progressive Agent guidance | Skill references | Library/Agent prompts | Tool descriptions | Tool descriptions |
315
+ | Action verification | Typed evidence + fresh state | State/history checks | Fresh snapshots | Manual inspection |
316
+ | Async recovery | Wait + command status + request IDs | Agent/controller lifecycle | Host/MCP lifecycle | Host/MCP lifecycle |
317
+ | Local media results | Paths + metadata | Library objects/paths | MCP image/resource results | MCP results |
318
+ | Network interception | Observe, block, mock, headers | Limited by controller | Supported | Supported |
319
+
320
+ Browser Pilot adopts several strong browser-use ideas without adopting its
321
+ Python Agent runtime: adaptive DOM/accessibility observation, fresh state after
322
+ actions, input readback, obstruction checks, stale-state invalidation,
323
+ watchdogs, and bounded outputs. The Agent interface remains the CLI.
324
+
325
+ ## Product Embedding
326
+
327
+ Bundle the native CLI or `browser-pilot-cli` plus a pinned Node runtime. Verify
328
+ the versioned release index and checksums. Put the bundled command directory on
329
+ the Agent's `PATH`, install the complete skill, and inject:
330
+
331
+ ```text
332
+ BROWSER_PILOT_CLIENT_KEY=<stable logical Agent identity>
333
+ BROWSER_PILOT_OUTPUT_DIR=<absolute task-owned directory>
458
334
  ```
459
335
 
460
- ## Testing
336
+ Do not map every CLI command into a native tool. The Agent already has a shell
337
+ tool, the skill controls context growth, and `bp --help` provides runtime
338
+ discovery. Agent-managed and product-bundled installations therefore exercise
339
+ the exact same implementation and guidance.
461
340
 
462
- Local release tests are deterministic. Real-site checks run separately as
463
- non-blocking canaries so a third-party outage cannot fail the release gate:
341
+ Each release ships a plugin archive, native archives, checksums, and a release
342
+ index binding the CLI version, compatible skill range, protocol range, and
343
+ supported assets. No `darwin-x64` artifact is produced.
344
+
345
+ ## Testing
464
346
 
465
347
  ```bash
466
- npm test # unit + local core, compat, and network gates
467
- npm run test:capabilities # isolated-Chrome quantitative capability gate
468
- npm run test:host-acceptance # two embedded hosts + CLI + Artifacts + reconnect
469
- npm run test:distribution # packed global/local/product-bundled launch modes
470
- npm run test:canary # real-site drift report; non-blocking
471
- npm run test:canary:strict # fail on drift or unavailability
472
- npm run test:integration # compatibility alias for strict canaries
473
- npm run test:all # release gates plus non-blocking canaries
348
+ npm run test:unit # TypeScript build and all Node tests
349
+ npm test # unit tests plus local Playwright gates
350
+ npm run test:capabilities # isolated-Chrome quantitative capability gate
351
+ npm run test:distribution # packed global/local/bundled CLI verification
352
+ npm run test:canary # non-blocking real-site drift report
353
+ npm run test:canary:strict
474
354
  ```
475
355
 
476
- Automated browser gates, including Playwright and stdio conformance, start their
477
- own headless Chrome with a temporary profile, random debugging port, and
478
- isolated Broker home. Teardown stops the Broker and Chrome and removes those
479
- files. Tests fail closed if that isolation is absent; they never attach to the
480
- user's Chrome by default. A maintainer can explicitly opt into a destructive
481
- manual compatibility run against the user's browser by setting
482
- `BROWSER_PILOT_TEST_USER_CHROME=1`.
356
+ Automated browser tests use isolated temporary Chrome profiles and Broker homes.
357
+ They never attach to the user's Chrome unless a maintainer explicitly opts into
358
+ a controlled manual test.
483
359
 
484
- Maintainers can build and verify a native artifact with an official,
485
- SEA-capable Node 22.17.0 runtime:
360
+ Build and verify a native artifact with an SEA-capable Node runtime:
486
361
 
487
362
  ```bash
488
363
  BROWSER_PILOT_SEA_NODE=/absolute/path/to/node npm run build:standalone
@@ -490,35 +365,21 @@ npm run verify:standalone
490
365
  npm run package:standalone
491
366
  ```
492
367
 
493
- For a new release, bump only the root package version through npm. The npm
494
- `version` lifecycle synchronizes every active distribution manifest from that
495
- source of truth:
368
+ The root `package.json` version is the release source of truth. The npm version
369
+ lifecycle synchronizes the lockfile, plugin manifests, skill compatibility
370
+ range, and marketplace entry:
496
371
 
497
372
  ```bash
498
373
  npm version "$BROWSER_PILOT_RELEASE_VERSION" --no-git-tag-version
499
374
  npm run release:check-version
500
- node scripts/verify-release-version.mjs
501
375
  ```
502
376
 
503
- The sync step updates the lockfile root, Agent plugin version and CLI peer
504
- range, Claude plugin/marketplace manifests, and skill `compatibility.json`.
505
- The current release becomes the skill's tested/minimum CLI version while the
506
- upper bound remains the next major version. Browser Pilot keeps public one-shot
507
- commands backward compatible within a major line, including additive `0.x`
508
- releases before 1.0. Historical release notes and version-specific fixtures are
509
- never rewritten.
510
-
511
- The canary report is written to
512
- `test-results/real-site-canary/report.json`. It distinguishes semantic drift,
513
- third-party unavailability, and runner errors. Set
514
- `BROWSER_PILOT_CANARY_REPORT` to use another report path.
515
-
516
377
  ## Requirements
517
378
 
518
- - Chrome 144+ / Edge / Brave (any Chromium-based browser)
519
- - Node.js >= 18
520
- - Remote debugging enabled (`chrome://inspect/#remote-debugging`)
521
- - Native archives: Apple Silicon macOS, x64 Linux, or x64 Windows; no Intel Mac
379
+ - Chrome 144+, Edge, Brave, or another supported Chromium browser
380
+ - Node.js 18 or newer for the npm distribution
381
+ - Remote debugging enabled at `chrome://inspect/#remote-debugging`
382
+ - Apple Silicon macOS, x64 Linux, or x64 Windows for native releases
522
383
 
523
384
  ## License
524
385