browser-pilot-cli 0.4.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +267 -406
- package/dist/cli.js +1136 -2564
- package/dist/daemon.js +131 -173
- package/docs/architecture/browser-pilot-platform-spec.md +443 -1047
- package/docs/plans/browser-capability-evolution.md +6 -6
- package/docs/plans/profile-context-routing.md +1 -1
- package/docs/plans/universal-agent-integration.md +121 -348
- package/docs/releases/v0.5.0.md +93 -0
- package/docs/releases/v0.5.1.md +30 -0
- package/package.json +2 -8
- package/docs/integration/stdio-bridge.md +0 -694
- package/docs/integration/stdio-conformance.md +0 -51
- package/scripts/run-stdio-conformance.mjs +0 -698
|
@@ -1,1078 +1,474 @@
|
|
|
1
1
|
# Browser Pilot Platform Specification
|
|
2
2
|
|
|
3
|
-
Status:
|
|
4
|
-
Baseline: `0c5661ccb133b2c9feda15732892f66ce947f232`
|
|
5
|
-
Last updated: 2026-07-25
|
|
6
|
-
|
|
7
|
-
This document is the source of truth for turning Browser Pilot from a
|
|
8
|
-
single-agent CLI with global state into a user-level browser control runtime
|
|
9
|
-
that any local Agent can install or embed. The two execution plans in
|
|
10
|
-
`docs/plans/` derive from this specification.
|
|
11
|
-
|
|
12
|
-
## Purpose and Reader
|
|
13
|
-
|
|
14
|
-
This specification is for Browser Pilot maintainers, Agent-product integrators,
|
|
15
|
-
security reviewers, and test authors. It defines observable behavior and
|
|
16
|
-
compatibility boundaries before implementation. Internal component names near
|
|
17
|
-
the end are implementation boundaries required to preserve those behaviors;
|
|
18
|
-
they are not third-party APIs.
|
|
19
|
-
|
|
20
|
-
## Evidence and Assumptions
|
|
21
|
-
|
|
22
|
-
Evidence from the baseline repository:
|
|
23
|
-
|
|
24
|
-
- The npm package currently publishes CLI binaries but no library exports.
|
|
25
|
-
- High-level actions, output formatting, file writes, and process exits are
|
|
26
|
-
combined in `src/cli.ts`.
|
|
27
|
-
- `src/state.ts` persists one active target, CDP session, and frame context.
|
|
28
|
-
- `src/snapshot.ts` persists one target's refs in a global file.
|
|
29
|
-
- `src/daemon.ts` exposes arbitrary CDP and keeps auth, network, dialog, and
|
|
30
|
-
popup state globally.
|
|
31
|
-
- Existing fixtures prove the current browser operations are worth preserving.
|
|
32
|
-
|
|
33
|
-
Evidence from the browser-use comparison is recorded in
|
|
34
|
-
`docs/plans/browser-capability-evolution.md` at the pinned research commit.
|
|
35
|
-
|
|
36
|
-
Assumptions used by this specification:
|
|
37
|
-
|
|
38
|
-
- Supported Agent runtimes can spawn a local child process and read protected
|
|
39
|
-
local Artifact files.
|
|
40
|
-
- Browser Pilot and embedded Agents execute under the same OS user account.
|
|
41
|
-
- Chrome continues to require an explicit user action to enable or authorize
|
|
42
|
-
remote debugging.
|
|
43
|
-
- An Agent product can map its own active lifecycle to Workspace and Lease
|
|
44
|
-
without Broker knowledge of that product's domain nouns.
|
|
45
|
-
|
|
46
|
-
## Decision Summary
|
|
47
|
-
|
|
48
|
-
- **DEC-1:** Browser Pilot is an Agent-neutral, per-OS-user Browser Broker. It
|
|
49
|
-
is not a Tenon component and does not model conversations, runs, tasks, or
|
|
50
|
-
workers.
|
|
51
|
-
- **DEC-2:** Root Chrome DevTools Protocol (CDP) is the only browser connector.
|
|
52
|
-
Browser extensions are never required or supported.
|
|
53
|
-
- **DEC-3:** The public product surface is one official CLI executable. It
|
|
54
|
-
supports human one-shot commands and a persistent `bridge --stdio` machine
|
|
55
|
-
mode. Native SDK and MCP surfaces are out of scope.
|
|
56
|
-
- **DEC-4:** Existing one-shot `bp` commands remain a first-class integration
|
|
57
|
-
path. Embedded products bundle the same executable and use the stdio bridge.
|
|
58
|
-
- **DEC-5:** The production machine interface exposes controlled browser tools,
|
|
59
|
-
not arbitrary CDP forwarding. Raw CDP is internal. JavaScript evaluation is
|
|
60
|
-
an explicit developer capability.
|
|
61
|
-
- **DEC-6:** Invoking Browser Pilot grants the Agent full control of all
|
|
62
|
-
eligible current and future tabs in the selected BrowserInstance. Browser
|
|
63
|
-
Pilot does not implement task-intent approval, selected-tab grants, or
|
|
64
|
-
per-action confirmation. An Agent host may remove operations when launching
|
|
65
|
-
the Broker, and owns any higher-level approval UX it wants.
|
|
66
|
-
- **DEC-7:** Browser profile state is shared because controlled targets run in
|
|
67
|
-
the user's browser profile. Target assignments, commands, refs, rules, and
|
|
68
|
-
artifacts are isolated by Broker objects.
|
|
69
|
-
|
|
70
|
-
## Objective, Constraints, and Selected Target
|
|
71
|
-
|
|
72
|
-
- **OBJ-1:** Any local Agent that can execute a process must be able to control
|
|
73
|
-
the user's real browser reliably, while preserving the user's profiles,
|
|
74
|
-
logins, cookies, and installed extensions.
|
|
75
|
-
- **OBJ-2:** Multiple unrelated Agent products must be able to use Browser Pilot
|
|
76
|
-
concurrently without overwriting each other's target, frame, ref, auth, or
|
|
77
|
-
network state.
|
|
78
|
-
- **OBJ-3:** Agents that already install and call `bp` themselves must continue
|
|
79
|
-
to work after the platform migration.
|
|
80
|
-
- **Minimum acceptable outcome:** The compatibility requirements in this spec
|
|
81
|
-
all pass. A partial SDK, Tenon-only adapter, or globally shared target state
|
|
82
|
-
does not count.
|
|
83
|
-
- **Clean-slate answer:** A signed, self-contained executable with a local
|
|
84
|
-
Broker, versioned stdio protocol, complete browser control, strong runtime
|
|
85
|
-
isolation, and a conformance kit.
|
|
86
|
-
- **Selected brownfield target:** Evolve the current TypeScript/npm CLI into
|
|
87
|
-
that architecture without replacing proven CDP capabilities or breaking the
|
|
88
|
-
current command workflow.
|
|
89
|
-
|
|
90
|
-
### Constraints
|
|
91
|
-
|
|
92
|
-
- **CON-1 hard:** No browser extension may be used.
|
|
93
|
-
- **CON-2 hard:** All eligible ordinary Chrome tabs in the selected
|
|
94
|
-
BrowserInstance are visible and controllable. DevTools, unsupported internal
|
|
95
|
-
pages, and Browser Pilot's own internal targets remain excluded.
|
|
96
|
-
- **CON-3 hard:** Browser Pilot must not depend on Agent-specific lifecycle
|
|
97
|
-
concepts or model-provider APIs.
|
|
98
|
-
- **CON-4 hard:** Production clients cannot invoke arbitrary root CDP methods.
|
|
99
|
-
- **CON-5 hard:** Sensitive browser data and media must not be emitted to logs
|
|
100
|
-
or world-readable files.
|
|
101
|
-
- **CON-6 legacy:** Existing `bp` commands, JSON output, npm installation, and
|
|
102
|
-
the current Agent skill must remain usable during migration.
|
|
103
|
-
- **CON-7 platform:** Without an extension, root CDP exposes the browser to any
|
|
104
|
-
process that can independently reach its debugging endpoint. Broker
|
|
105
|
-
isolation protects Broker clients from each other; it is not an OS sandbox.
|
|
106
|
-
- **CON-8 scope:** The universal integration target is local Agents that can
|
|
107
|
-
spawn a process and access local files. Hosted Agents without a local
|
|
108
|
-
execution bridge are not supported by CLI-only distribution.
|
|
109
|
-
|
|
110
|
-
### Rejected Alternatives
|
|
111
|
-
|
|
112
|
-
- **OPT-1 MCP as the canonical interface:** Rejected because it creates a
|
|
113
|
-
second public product surface and runtime dependency. An Agent adapter may
|
|
114
|
-
translate the stdio bridge to MCP outside Browser Pilot.
|
|
115
|
-
- **OPT-2 Native SDKs:** Rejected because every language binding would create a
|
|
116
|
-
versioned compatibility obligation. The executable protocol is the SDK.
|
|
117
|
-
- **OPT-3 One-shot CLI only:** Rejected because it cannot efficiently deliver
|
|
118
|
-
events, cancellation, leases, or repeated tool calls to an embedded Agent.
|
|
119
|
-
- **OPT-4 Agent-specific embedding:** Rejected because Tenon and OpenClaw are
|
|
120
|
-
clients and conformance fixtures, not Broker domain concepts.
|
|
121
|
-
|
|
122
|
-
### Revisit Triggers
|
|
123
|
-
|
|
124
|
-
Reconsider MCP or a remote protocol only if Browser Pilot must support Agents
|
|
125
|
-
that cannot spawn a local process. Reconsider SDKs only if the stdio protocol
|
|
126
|
-
cannot provide required performance or type safety after measured use.
|
|
127
|
-
|
|
128
|
-
## Preserved and Changed Behavior
|
|
129
|
-
|
|
130
|
-
### Preserved
|
|
131
|
-
|
|
132
|
-
- Agent-managed installation via npm and direct `bp` invocation.
|
|
133
|
-
- Chrome remote-debugging discovery and connection.
|
|
134
|
-
- A distinct visible Pilot window using the user's browser profile.
|
|
135
|
-
- AX snapshots and numbered refs.
|
|
136
|
-
- Navigation, click, type, keyboard, upload, tabs, frames, dialogs, HTTP auth,
|
|
137
|
-
screenshots, PDF, cookies, and network tooling.
|
|
138
|
-
- Human-readable terminal output and JSON output for scripts.
|
|
139
|
-
- The existing skill and Playwright fixture coverage.
|
|
140
|
-
|
|
141
|
-
### Changed
|
|
142
|
-
|
|
143
|
-
- CLI handlers become presentation adapters over reusable Broker services.
|
|
144
|
-
- Global active target/session/frame state becomes isolated Workspace and Lease
|
|
145
|
-
state.
|
|
146
|
-
- The single refs file becomes observation-scoped Broker state.
|
|
147
|
-
- The daemon's arbitrary `/cdp` endpoint becomes private implementation wiring.
|
|
148
|
-
- Dialogs become explicit events and decisions; they are not auto-accepted.
|
|
149
|
-
- Popup adoption requires a verified owned opener chain.
|
|
150
|
-
- Existing user tabs and newly opened tabs are included automatically in the
|
|
151
|
-
Workspace inventory; invoking the tool is the authorization boundary.
|
|
152
|
-
- Auth, network rules, request journals, and artifacts are scoped and cleaned up
|
|
153
|
-
by the owning Workspace or Lease.
|
|
154
|
-
- Screenshots and PDFs become mode-restricted, owner-scoped Artifact results
|
|
155
|
-
with quotas and expiry. The human CLI may export an Artifact to a
|
|
156
|
-
caller-selected path.
|
|
157
|
-
|
|
158
|
-
## Product Model
|
|
159
|
-
|
|
160
|
-
### ClientPrincipal
|
|
161
|
-
|
|
162
|
-
A stable identity for an installed Agent product or direct CLI user. It owns
|
|
163
|
-
Workspaces and their transient resources. It is used for correlation and
|
|
164
|
-
isolation, not as proof that an Agent's semantic decision is correct. A
|
|
165
|
-
Principal is not a conversation or task.
|
|
166
|
-
|
|
167
|
-
### ClientConnection
|
|
168
|
-
|
|
169
|
-
One live `bridge --stdio` process connection. It is authenticated by the local
|
|
170
|
-
Broker and associated with exactly one ClientPrincipal after initialization.
|
|
171
|
-
|
|
172
|
-
### BrowserInstance
|
|
173
|
-
|
|
174
|
-
One running, discoverable Chromium-family browser endpoint identified by a
|
|
175
|
-
process-stable identity, product, user-data root, debugging endpoint, and
|
|
176
|
-
connection generation. One endpoint may expose several simultaneously live
|
|
177
|
-
Chrome Profile contexts; those are not separate BrowserInstances.
|
|
178
|
-
|
|
179
|
-
### ProfileContext
|
|
180
|
-
|
|
181
|
-
One live regular browser context inside a BrowserInstance. Its public
|
|
182
|
-
`profileContextId` is opaque and scoped to the current browser connection
|
|
183
|
-
generation; the raw CDP browser-context ID is internal. A ProfileContext has a
|
|
184
|
-
connection-scoped neutral label, structured identity status and optional
|
|
185
|
-
verified Profile/account metadata, bounded
|
|
186
|
-
representative tabs, and current tab counts. Passive discovery never opens a
|
|
187
|
-
tab to infer Profile or account identity. Protocol 1.3 identity is an explicit visible
|
|
188
|
-
operation: it reads Chrome's reported Profile path, verifies it against the
|
|
189
|
-
BrowserInstance user-data root, then maps bounded Local State metadata to
|
|
190
|
-
structured Profile/account fields. Reconnect invalidates every ProfileContext
|
|
191
|
-
ID and cached identity.
|
|
192
|
-
|
|
193
|
-
### BrowserWorkspace
|
|
194
|
-
|
|
195
|
-
A logical browser working set owned by one ClientPrincipal. It owns one or more
|
|
196
|
-
ManagedTabSets, an optional selected ProfileContext, and transient scoped state.
|
|
197
|
-
Workspaces live in Broker memory by default and are recreated after Broker
|
|
198
|
-
restart; Agent adapters map their own active lifecycle to this object. Profile
|
|
199
|
-
selection is never persisted as a global browser default.
|
|
200
|
-
|
|
201
|
-
### ManagedTabSet
|
|
202
|
-
|
|
203
|
-
The logical collection of Broker-created tabs used for isolated Agent work. It
|
|
204
|
-
is normally backed by a dedicated visible Pilot window because Browser Pilot
|
|
205
|
-
does not use extension-only native tab-group APIs. New task navigation and
|
|
206
|
-
owned popups stay in this set by default. A ManagedTabSet is permanently bound
|
|
207
|
-
to at most one ProfileContext; a Workspace that creates managed work in multiple
|
|
208
|
-
Profiles owns one set per Profile. Workspace cleanup may close these sets, but
|
|
209
|
-
never user-owned tabs outside them.
|
|
210
|
-
|
|
211
|
-
### ControlLease
|
|
212
|
-
|
|
213
|
-
A time-bounded command session in a Workspace. It carries negotiated
|
|
214
|
-
capabilities, heartbeat expiry, and explicit target control assignments. Agent
|
|
215
|
-
adapters map their own active work lifecycle to this object.
|
|
3
|
+
Status: active architecture for the next release line.
|
|
216
4
|
|
|
217
|
-
|
|
5
|
+
## 1. Product Definition
|
|
218
6
|
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
Browser Pilot approval lifecycle, and it cannot be expanded through a tool call
|
|
223
|
-
inside the running bridge.
|
|
7
|
+
Browser Pilot gives a local AI Agent controlled access to the user's eligible
|
|
8
|
+
Chromium tabs through a command-line interface. It reuses the user's live
|
|
9
|
+
profiles, logins, cookies, and extensions without a browser extension.
|
|
224
10
|
|
|
225
|
-
|
|
11
|
+
The product is not an Agent runtime. It does not decide user intent, add
|
|
12
|
+
per-action approvals, own the model conversation, or expose a second native
|
|
13
|
+
tool protocol.
|
|
226
14
|
|
|
227
|
-
|
|
228
|
-
or `user_tab`, and it records its ProfileContext and an optional ManagedTabSet.
|
|
229
|
-
A user tab retains user
|
|
230
|
-
ownership: releasing a Workspace removes Broker mappings without closing the
|
|
231
|
-
tab. Every result identifies its Workspace, Lease, opaque controlled-target ID,
|
|
232
|
-
URL, origin, and browser connection generation. Raw CDP target IDs are not a
|
|
233
|
-
public identifier.
|
|
234
|
-
|
|
235
|
-
### Observation
|
|
15
|
+
## 2. Architectural Decision
|
|
236
16
|
|
|
237
|
-
|
|
238
|
-
handles are `observationId + ref`. Observations are never shared between
|
|
239
|
-
Workspaces.
|
|
17
|
+
There is one public Agent integration path:
|
|
240
18
|
|
|
241
|
-
|
|
19
|
+
```text
|
|
20
|
+
Agent
|
|
21
|
+
-> Browser Pilot skill
|
|
22
|
+
-> host's existing shell/command runner
|
|
23
|
+
-> short-lived bp CLI process
|
|
24
|
+
-> shared per-user Browser Pilot Broker
|
|
25
|
+
-> supervised CDP connection
|
|
26
|
+
-> Chrome
|
|
27
|
+
```
|
|
242
28
|
|
|
243
|
-
|
|
244
|
-
event. A hint has a stable `code`, `source`, `confidence`, and
|
|
245
|
-
`recommendedAction`, plus code-specific public data. Hints report browser
|
|
246
|
-
evidence and a useful next inspection strategy; they never approve an action,
|
|
247
|
-
assert that the task succeeded, or replace fresh Observation and event state.
|
|
29
|
+
Browser Pilot does not publish or support:
|
|
248
30
|
|
|
249
|
-
|
|
31
|
+
- a browser extension;
|
|
32
|
+
- an MCP server;
|
|
33
|
+
- a language-specific Native SDK;
|
|
34
|
+
- Browser Pilot-specific native Agent tools;
|
|
35
|
+
- a public JSON-RPC or stdio adapter;
|
|
36
|
+
- imports from `src/*`;
|
|
37
|
+
- direct consumer access to Broker sockets or locator files.
|
|
250
38
|
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
39
|
+
This decision applies equally to Agent-managed installs and products that
|
|
40
|
+
bundle Browser Pilot. A product makes the CLI available on the Agent command
|
|
41
|
+
environment's `PATH`, installs the same skill, and injects stable environment.
|
|
254
42
|
|
|
255
|
-
|
|
43
|
+
## 3. Public Contract
|
|
256
44
|
|
|
257
|
-
|
|
258
|
-
Artifacts have metadata, sensitivity, ownership, size limits, retention state,
|
|
259
|
-
and expiry.
|
|
45
|
+
The supported public contract consists of:
|
|
260
46
|
|
|
261
|
-
|
|
47
|
+
1. The `bp` and `browser-pilot` executable names.
|
|
48
|
+
2. Command syntax and `--help` output.
|
|
49
|
+
3. JSON success and failure results.
|
|
50
|
+
4. Stable machine error codes.
|
|
51
|
+
5. Local file results and their metadata.
|
|
52
|
+
6. Environment variables documented in this repository.
|
|
53
|
+
7. Skill files and their declared CLI compatibility range.
|
|
54
|
+
8. Versioned native archives, npm package, checksums, and release index.
|
|
262
55
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
Command terminal state and explicit reconnect cleanup remain replayable with
|
|
267
|
-
their historical generation.
|
|
56
|
+
Internal Broker methods, schemas, IDs, sockets, files, CDP sessions, and
|
|
57
|
+
process roles are not public APIs. They may evolve without creating a second
|
|
58
|
+
integration surface as long as the CLI contract remains compatible.
|
|
268
59
|
|
|
269
|
-
##
|
|
60
|
+
## 4. Host Integration Contract
|
|
270
61
|
|
|
271
|
-
|
|
272
|
-
and the compatible `bp` alias.
|
|
62
|
+
A shell-capable Agent host needs only:
|
|
273
63
|
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
artifact export|release
|
|
281
|
-
browsers|connect|disconnect|open|snapshot|click|type|keyboard|press
|
|
282
|
-
read|eval|upload|screenshot|pdf|cookies|frame|auth|tabs|tab|close|net
|
|
283
|
-
```
|
|
64
|
+
- a normal local command execution tool;
|
|
65
|
+
- the complete Browser Pilot skill;
|
|
66
|
+
- a compatible `bp` executable on the command environment's `PATH`;
|
|
67
|
+
- a stable `BROWSER_PILOT_CLIENT_KEY` per independent Agent;
|
|
68
|
+
- an absolute task-owned `BROWSER_PILOT_OUTPUT_DIR` when file results are used;
|
|
69
|
+
- ordinary access to returned task files.
|
|
284
70
|
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
2. Project-local installation and `npx browser-pilot` invocation.
|
|
289
|
-
3. Product-bundled official executable launched by an Agent adapter.
|
|
290
|
-
|
|
291
|
-
Embedded products must not import `src/*`, depend on daemon socket details, or
|
|
292
|
-
parse human-readable output.
|
|
293
|
-
|
|
294
|
-
The npm distribution contains bundled JavaScript entry points for the CLI,
|
|
295
|
-
Broker, and managed-target janitor and requires Node 18 or newer. The native
|
|
296
|
-
distribution is one Node SEA executable containing the runtime and JavaScript
|
|
297
|
-
dependencies. It re-executes itself with reserved private role arguments for
|
|
298
|
-
Broker and janitor processes; it neither downloads a runtime nor falls back to
|
|
299
|
-
system Node. Both layouts derive executable installation identity from the
|
|
300
|
-
public entry point, so private child roles do not create false version
|
|
301
|
-
boundaries.
|
|
302
|
-
|
|
303
|
-
Native artifacts are built and verified on their target OS and architecture.
|
|
304
|
-
Each archive includes an actual-state signature descriptor, file sizes and
|
|
305
|
-
SHA-256 hashes, runtime/dependency licenses, and an archive checksum. macOS uses
|
|
306
|
-
Developer ID signing with hardened runtime and explicit JIT entitlements when
|
|
307
|
-
credentials exist, otherwise ad-hoc signing. Windows uses Authenticode when a
|
|
308
|
-
certificate exists, otherwise records unsigned state. Linux records unsigned
|
|
309
|
-
state. Build or release automation must fail on incomplete credentials and
|
|
310
|
-
must never describe an artifact as signed when signing did not run.
|
|
311
|
-
|
|
312
|
-
The supported native matrix is `darwin-arm64`, `linux-x64`, and `win32-x64`;
|
|
313
|
-
Intel Mac is not supported. Each release also publishes a versioned Agent
|
|
314
|
-
plugin/skill archive and checksum. A deterministic release index binds those
|
|
315
|
-
assets to the npm package version, exact compatible skill/CLI version, and
|
|
316
|
-
runtime protocol range. GitHub Release publication waits for successful npm
|
|
317
|
-
publication so consumers cannot observe a nominal release with only one half
|
|
318
|
-
of that distribution contract.
|
|
319
|
-
|
|
320
|
-
### Broker Locator and Startup
|
|
321
|
-
|
|
322
|
-
The executable, not an embedding Host, resolves the per-user Broker endpoint.
|
|
323
|
-
On Unix it uses an owner-only domain socket and falls back to a deterministic
|
|
324
|
-
short owner-specific runtime directory when the state path would exceed
|
|
325
|
-
portable socket limits. On Windows it uses a named pipe derived from the OS
|
|
326
|
-
user identity. Persistent state defaults to the Browser Pilot user directory
|
|
327
|
-
(`%LOCALAPPDATA%\\Browser Pilot` on Windows). An explicit absolute
|
|
328
|
-
`BROWSER_PILOT_HOME` is the cross-platform installation-level isolation
|
|
329
|
-
override. It is never selected automatically.
|
|
330
|
-
|
|
331
|
-
State/runtime directories are mode `0700` and regular metadata files are mode
|
|
332
|
-
`0600` where POSIX modes apply. The socket is mode `0600`; Windows relies on the
|
|
333
|
-
creating user's process-token DACL for its local named pipe. Symlinked final
|
|
334
|
-
directories and metadata, foreign ownership, unexpected file types, and
|
|
335
|
-
overly-permissive metadata are rejected.
|
|
336
|
-
|
|
337
|
-
The versioned locator records PID, endpoint/transport, start time, Broker
|
|
338
|
-
process identity, service version, executable installation identity/path, and
|
|
339
|
-
the supported protocol range. A separate owner-only `broker-versions.json`
|
|
340
|
-
keeps only the current and immediately previous executable metadata for
|
|
341
|
-
diagnosis and explicit rollback; it does not preserve an executable or any
|
|
342
|
-
browser, target, Workspace, Lease, ref, credential, rule, or command state. An
|
|
343
|
-
atomic owner-recorded startup lock surrounds discovery and daemon launch. Every
|
|
344
|
-
contender health-checks before and after the lock; therefore exactly one
|
|
345
|
-
process starts the Broker and all others reuse it. Broker startup and discovery
|
|
346
|
-
are passive and do not open a browser WebSocket.
|
|
347
|
-
A lock whose owner is dead can be reclaimed; a live owner is never displaced
|
|
348
|
-
merely because startup is slow. During Broker startup, the daemon writes an
|
|
349
|
-
owner-only `starting` process record. Chrome authorization begins only after an
|
|
350
|
-
explicit `browser.connect`; all concurrent callers share its single in-flight
|
|
351
|
-
connection attempt. A dead
|
|
352
|
-
locator/socket is removed without signaling its recorded PID. If a ready PID is
|
|
353
|
-
alive but its endpoint is unresponsive, Browser Pilot returns structured
|
|
354
|
-
restart remediation and never silently kills or replaces the process.
|
|
355
|
-
|
|
356
|
-
## Primary Flows
|
|
357
|
-
|
|
358
|
-
### FLOW-1 Agent-managed one-shot use
|
|
359
|
-
|
|
360
|
-
An Agent installs Browser Pilot globally or in its project and runs `bp connect`.
|
|
361
|
-
The CLI starts or reuses the compatible per-user Broker, creates its
|
|
362
|
-
compatibility Workspace and Lease, then explicitly requests the browser
|
|
363
|
-
connection. Other browser commands require that connection and never trigger
|
|
364
|
-
authorization implicitly.
|
|
365
|
-
Existing and future eligible user tabs are available immediately. The command
|
|
366
|
-
returns the existing JSON shape during migration. Every non-TTY failure,
|
|
367
|
-
including command parsing and local argument validation, includes a stable
|
|
368
|
-
machine code and `retryable` flag in addition to compatible human guidance.
|
|
369
|
-
One-shot processes
|
|
370
|
-
derive a stable Connection identity from `BROWSER_PILOT_CLIENT_KEY` or
|
|
371
|
-
`--client-key`, then use an idempotently keyed Workspace and renewable Lease
|
|
372
|
-
with a maximum five-minute TTL. Independent Agents use distinct stable keys;
|
|
373
|
-
the default key preserves single-Agent compatibility. Normal process exit leaves that transient
|
|
374
|
-
daemon-memory state available to the next command; expiry invalidates refs and
|
|
375
|
-
releases target control. No target, frame, session, Observation, ref, auth, or
|
|
376
|
-
network mapping is persisted to disk. `bp disconnect` explicitly releases the
|
|
377
|
-
compatibility Workspace and closes only its managed targets. It stops the
|
|
378
|
-
daemon only when called by the matching executable installation and no other
|
|
379
|
-
active Lease or embedded Connection remains; otherwise it returns
|
|
380
|
-
`protocol_incompatible` or
|
|
381
|
-
`broker_in_use` without replacing or terminating the Broker. JavaScript dialogs
|
|
382
|
-
remain pending and are handled explicitly with `bp dialogs` and `bp dialog`;
|
|
383
|
-
Workspace isolation prevents access to other clients' dialogs.
|
|
384
|
-
|
|
385
|
-
### FLOW-2 Product-embedded use
|
|
386
|
-
|
|
387
|
-
An Agent product bundles or resolves the official executable and starts
|
|
388
|
-
`browser-pilot bridge --stdio`. It initializes, negotiates protocol and
|
|
389
|
-
capabilities, creates or resumes a Workspace, acquires a Lease, discovers tools,
|
|
390
|
-
calls `browser.connect` when the selected browser is not ready, and maps the
|
|
391
|
-
remaining tools into its own runtime. Bridge launch, initialization, discovery,
|
|
392
|
-
and Workspace creation stay passive. It calls tools, consumes ordered
|
|
393
|
-
events, converts Artifact files into native media content, heartbeats while
|
|
394
|
-
active, then releases the Lease. A process crash is recovered by Lease expiry.
|
|
395
|
-
|
|
396
|
-
### FLOW-3 Browser disconnect and recovery
|
|
397
|
-
|
|
398
|
-
The Broker emits connection loss, stops dispatching target commands, and marks
|
|
399
|
-
commands whose effects are uncertain as `unknown_outcome`. It never reconnects
|
|
400
|
-
on a timer. A client explicitly calls `browser.connect` when recovery is
|
|
401
|
-
appropriate. After that succeeds, the Broker
|
|
402
|
-
creates a new browser connection generation, invalidates old CDP sessions,
|
|
403
|
-
ControlledTarget mappings, frames, and Observations, and emits structured recovery
|
|
404
|
-
state. The daemon rediscovers only the originally selected browser profile; it
|
|
405
|
-
does not silently switch to another running browser. Consumers rebuild inventory
|
|
406
|
-
with `browser.tabs.list`. Browser Pilot never replays a mutating command
|
|
407
|
-
automatically.
|
|
408
|
-
|
|
409
|
-
### FLOW-4 Concurrent clients
|
|
410
|
-
|
|
411
|
-
Two unrelated Principals create separate Workspaces and Pilot targets. Their
|
|
412
|
-
actions may execute concurrently on different target actors. Any attempt to use
|
|
413
|
-
the other Principal's target, ref, rule, Artifact, or Workspace fails closed.
|
|
414
|
-
Releasing or crashing one client does not close or mutate the other's targets.
|
|
415
|
-
|
|
416
|
-
### FLOW-5 Existing and future user tabs
|
|
417
|
-
|
|
418
|
-
When a Host exposes Browser Pilot to an Agent, that invocation authorizes the
|
|
419
|
-
Agent to use the selected BrowserInstance. `tabs.list` immediately returns the
|
|
420
|
-
Workspace's ManagedTabSets plus all eligible user tabs, tagged by origin. Tabs
|
|
421
|
-
opened later appear on the next inventory refresh. The Agent may switch to and
|
|
422
|
-
operate any listed tab, including a form the user opened before invoking it.
|
|
423
|
-
|
|
424
|
-
Browser Pilot does not ask the Agent to request access and does not display a
|
|
425
|
-
grant UI. If a product needs confirmation or restricted tool exposure, the
|
|
426
|
-
Agent host implements that policy before launching the bridge or maps it to a
|
|
427
|
-
launch-time BrowserControlPolicy. Releasing a Lease or Workspace removes
|
|
428
|
-
transient mappings and rules but leaves user tabs open. Closing a user tab
|
|
429
|
-
always requires an explicit target-specific command; bulk cleanup applies only
|
|
430
|
-
to ManagedTabSets.
|
|
431
|
-
|
|
432
|
-
### FLOW-6 Multiple live Chrome Profiles
|
|
433
|
-
|
|
434
|
-
`browser.tabs.list` inventories eligible tabs from every Profile context exposed
|
|
435
|
-
by the selected BrowserInstance. Existing targets are immediately controllable
|
|
436
|
-
and carry an opaque `profileContextId`. New managed work resolves a Profile from
|
|
437
|
-
an explicit context, the current Lease-selected target, the Workspace selection, or a
|
|
438
|
-
single available context, in that order. Multiple contexts without an anchor
|
|
439
|
-
return `profile_selection_required` before creating a target. The Agent host
|
|
440
|
-
asks the user which Profile to use; Browser Pilot does not add an approval UI.
|
|
441
|
-
|
|
442
|
-
`browser.profiles.list` is passive and returns bounded, connection-scoped
|
|
443
|
-
summaries. Protocol 1.3 `browser.profiles.identify` explicitly creates and
|
|
444
|
-
cleans temporary visible identity targets when account-aware identity is
|
|
445
|
-
needed. `browser.profiles.select` updates only the owning Workspace. A Chrome
|
|
446
|
-
reconnect invalidates old ProfileContext IDs and requires relisting and
|
|
447
|
-
reselection. Selection clears only the Lease's logical selected-target anchor; it
|
|
448
|
-
does not release control, activate a tab, or change Chrome focus. The exact
|
|
449
|
-
creation and compatibility algorithm is frozen in `docs/plans/profile-context-routing.md`.
|
|
450
|
-
|
|
451
|
-
## Machine Protocol
|
|
452
|
-
|
|
453
|
-
`bridge --stdio` uses JSON-RPC 2.0 over newline-delimited UTF-8 JSON. Each line
|
|
454
|
-
contains one complete message. Stdout is protocol-only. Logs go to stderr and
|
|
455
|
-
must redact secrets. The bridge terminates on malformed framing after returning
|
|
456
|
-
a protocol error when possible.
|
|
457
|
-
|
|
458
|
-
Required methods:
|
|
71
|
+
The host must not map every Browser Pilot command into a native tool. The skill
|
|
72
|
+
provides progressive disclosure, and CLI help provides exact runtime command
|
|
73
|
+
discovery.
|
|
459
74
|
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
workspaces/create|get|release
|
|
465
|
-
leases/create|heartbeat|release
|
|
466
|
-
commands/cancel|get
|
|
467
|
-
events/poll
|
|
468
|
-
artifacts/import|get|export|retain|release
|
|
469
|
-
shutdown
|
|
470
|
-
```
|
|
75
|
+
The host should pin the CLI version tested by its product release. The skill
|
|
76
|
+
declares a compatible range so a compatible user-installed command can also be
|
|
77
|
+
used. The host must update its executable, skill, compatibility metadata, and
|
|
78
|
+
checksums together.
|
|
471
79
|
|
|
472
|
-
|
|
473
|
-
Polling with a cursor is the authoritative recovery path after reconnect,
|
|
474
|
-
notification loss, or backpressure. Notifications and responses may interleave.
|
|
475
|
-
|
|
476
|
-
### Initialization
|
|
477
|
-
|
|
478
|
-
The client sends its product identity, instance identity, supported protocol
|
|
479
|
-
range, requested capabilities, launch mode, and optional protocol 1.1 transport
|
|
480
|
-
limits. The response includes:
|
|
481
|
-
|
|
482
|
-
- service and executable versions;
|
|
483
|
-
- selected protocol version;
|
|
484
|
-
- supported and granted capabilities;
|
|
485
|
-
- Broker process identity and connection ID;
|
|
486
|
-
- browser discovery and authorization state;
|
|
487
|
-
- transport and result size limits.
|
|
488
|
-
|
|
489
|
-
Initialization must fail with a structured incompatibility error when no
|
|
490
|
-
protocol overlap exists. A newer executable must not silently replace an
|
|
491
|
-
incompatible Broker that has live clients.
|
|
492
|
-
|
|
493
|
-
Protocol 1.0 clients use the service transport ceilings. Protocol 1.1 clients
|
|
494
|
-
may declare maximum message and result byte sizes; the Broker selects the lower
|
|
495
|
-
client/service value per Connection. Initialization itself uses fixed bootstrap
|
|
496
|
-
ceilings, and pipelined messages after initialize are not parsed until the
|
|
497
|
-
selected limits take effect. Artifact and journal capacities remain service
|
|
498
|
-
resource limits. Pending ordinary and out-of-band control calls have separate
|
|
499
|
-
bounds so overload cannot create an unbounded Promise queue or prevent dialog
|
|
500
|
-
and cancellation control from overtaking a blocked action.
|
|
501
|
-
|
|
502
|
-
Protocol 1.2 adds connection-generation-scoped ProfileContext discovery and
|
|
503
|
-
selection, Profile identity on tab inventory, and context-aware managed target
|
|
504
|
-
creation. Protocol 1.0/1.1 clients retain existing-target control and automatic
|
|
505
|
-
single-context behavior, but an ambiguous multi-context open fails rather than
|
|
506
|
-
silently selecting a Profile.
|
|
507
|
-
|
|
508
|
-
Protocol 1.3 adds explicit verified Profile/account identity. Browser candidate
|
|
509
|
-
results use `userDataRoot`, Profile results use structured identity status and
|
|
510
|
-
fields, and tab inventory uses `selected` for the Lease-local target instead of
|
|
511
|
-
the ambiguous `active`. Protocol 1.2 retains its legacy field vocabulary.
|
|
512
|
-
|
|
513
|
-
### Tool Contract
|
|
514
|
-
|
|
515
|
-
`tools/list` is the canonical machine-readable manifest. Every tool declares:
|
|
516
|
-
|
|
517
|
-
- stable name and description;
|
|
518
|
-
- JSON input schema;
|
|
519
|
-
- output schema and possible Artifact kinds;
|
|
520
|
-
- required capabilities;
|
|
521
|
-
- whether it may mutate browser state;
|
|
522
|
-
- cancellation and idempotency semantics;
|
|
523
|
-
- sensitivity classification.
|
|
524
|
-
|
|
525
|
-
The tool-level `sensitivity.input` and `sensitivity.output` arrays declare every
|
|
526
|
-
classification that may occur in that direction. Content-bearing input and
|
|
527
|
-
output schema nodes use `x-browser-pilot-sensitivity` to identify the specific
|
|
528
|
-
fields an adapter must taint when it maps values into model text, image, or file
|
|
529
|
-
content. The manifest validator rejects unknown, duplicate, or field-level
|
|
530
|
-
classifications missing from the tool-level declaration. Schema annotations do
|
|
531
|
-
not change the call or result value shape.
|
|
532
|
-
|
|
533
|
-
`tools/call` returns `{ command, result?, error? }`; each manifest output schema
|
|
534
|
-
describes the inner `result`. Callers may provide an opaque Command ID,
|
|
535
|
-
idempotency key, and relative deadline. `commands/get` and `commands/cancel`
|
|
536
|
-
accept a Command ID plus its Workspace ID when applicable. They authorize by
|
|
537
|
-
ClientPrincipal ownership so a replacement Connection can inspect known state
|
|
538
|
-
without inheriting the original Lease.
|
|
539
|
-
|
|
540
|
-
Action results include bounded typed evidence where Browser Pilot can observe
|
|
541
|
-
an effect. Ref clicks combine local control state and composed focus changes
|
|
542
|
-
with Broker-owned navigation, normalized document, dialog, and attributable
|
|
543
|
-
popup signals. `verified` means at least one supported effect was observed,
|
|
544
|
-
`mismatch` means an observable control did not reach its expected state, and
|
|
545
|
-
`unavailable` means dispatch completed without conclusive evidence. Evidence is
|
|
546
|
-
not a claim that the Agent's higher-level business goal succeeded.
|
|
547
|
-
Input evidence is discriminated as `type` or `keyboard`; press evidence uses the
|
|
548
|
-
focused backend node plus bounded control-state signatures and Broker-owned
|
|
549
|
-
signals. Upload evidence reads the selected file count and compares its browser
|
|
550
|
-
filename without returning the local source path.
|
|
551
|
-
|
|
552
|
-
Observation-producing tools also return `hints`. Observation-derived codes
|
|
553
|
-
cover explicit autocomplete surfaces, modal overlays, explicit filter controls,
|
|
554
|
-
and authentication surface transitions; action results may add a repeated
|
|
555
|
-
observable no-progress hint. Hint refs are numbered refs from that Observation
|
|
556
|
-
only. Page collection returns counts and refs, never page text, input values,
|
|
557
|
-
passwords, selectors, raw nodes, or CDP identifiers for this classification.
|
|
558
|
-
Blocked main-document responses and download lifecycle hints are event-derived.
|
|
559
|
-
|
|
560
|
-
`hints` is an additive schema field under protocol 1.0 and 1.1. The built-in
|
|
561
|
-
Browser tool implementation always returns the array, including an empty array;
|
|
562
|
-
the manifest leaves the field optional so compatible older executors remain
|
|
563
|
-
valid. Existing clients already ignore unknown response fields. No new
|
|
564
|
-
capability is negotiated because hints expose no operation and contain only
|
|
565
|
-
bounded derivatives of browser data already authorized by `observation.read`
|
|
566
|
-
or `event.read`.
|
|
567
|
-
|
|
568
|
-
Composite input actions pin the ControlledTarget, CDP session, selected frame,
|
|
569
|
-
loader, Document identity, and browser connection generation for the lifetime
|
|
570
|
-
of the action. After focus is established, `type` and `keyboard` also pin the
|
|
571
|
-
deep composed active element. Browser Pilot checks this identity before every
|
|
572
|
-
remaining semantic step, including select-all, deletion, each keyboard
|
|
573
|
-
character, and optional submit. A change before the first browser mutation is
|
|
574
|
-
a retryable `action_not_verified`; a change after any step was dispatched is
|
|
575
|
-
`unknown_outcome` with bounded `action`, `step`, `reason`, `dispatchedSteps`,
|
|
576
|
-
and `remainingStepsStopped` context. No remaining step is sent to the changed
|
|
577
|
-
page and the command is never replayed automatically.
|
|
578
|
-
|
|
579
|
-
The initial controlled tool families are browser discovery/status, Workspace
|
|
580
|
-
and Lease lifecycle, open/observe/read, click/type/keyboard/press,
|
|
581
|
-
capture/upload, tabs/frames, dialogs, auth/cookies, network, events, and
|
|
582
|
-
Artifacts. Raw CDP is never listed. `eval` requires `developer.eval`.
|
|
583
|
-
|
|
584
|
-
## Access and Isolation Rules
|
|
585
|
-
|
|
586
|
-
- **BR-1:** A public command can address a ControlledTarget only through its
|
|
587
|
-
Workspace and a valid ControlLease.
|
|
588
|
-
- **BR-2:** Managed targets enter a Workspace through Broker creation and a
|
|
589
|
-
verified managed-popup opener chain. Eligible user tabs enter through target
|
|
590
|
-
inventory. Public commands use opaque ControlledTarget IDs; Agent-supplied
|
|
591
|
-
raw CDP identifiers are never accepted as public handles.
|
|
592
|
-
- **BR-3:** Closing or releasing a Workspace may close its ManagedTabSets, but
|
|
593
|
-
must not close user tabs merely because they were visible or controlled. It
|
|
594
|
-
affects only its own mappings, rules, credentials, journals, and Artifacts.
|
|
595
|
-
- **BR-4:** Network and auth configuration is not global. Precedence and cleanup
|
|
596
|
-
are deterministic within the owning Workspace.
|
|
597
|
-
- **BR-5:** Each physical browser target serializes state-changing commands.
|
|
598
|
-
Reads that depend on document state join that same actor even when separate
|
|
599
|
-
Workspaces address the target through different opaque ControlledTarget IDs.
|
|
600
|
-
- **BR-6:** A physical target has at most one controlling Lease at a time.
|
|
601
|
-
Multiple Workspaces may inventory it under different opaque IDs, but a
|
|
602
|
-
conflicting acquisition reports `target_busy` and never silently steals
|
|
603
|
-
control. The controlling Lease may call `browser.tabs.release` with its own
|
|
604
|
-
opaque target ID. The Broker retires its target session and scoped state
|
|
605
|
-
before a different Lease can acquire control with that Lease's own opaque ID.
|
|
606
|
-
No handoff call accepts a foreign Lease or target ID.
|
|
607
|
-
- **BR-7:** Browser Pilot returns controlling-client identity in scoped
|
|
608
|
-
protocol results and events so a Host may present it in Host-owned UI. With no
|
|
609
|
-
extension, Browser Pilot does not claim it can modify Chrome's browser chrome,
|
|
610
|
-
and it never injects status borders, labels, or controls into page DOM.
|
|
611
|
-
- **BR-21:** Exposing the CLI or bridge to an Agent is the product-level
|
|
612
|
-
browser-control authorization boundary. Chrome's own remote-debugging
|
|
613
|
-
authorization is requested only by explicit `bp connect`/`browser.connect`.
|
|
614
|
-
Browser Pilot does not infer task intent, ask for per-tab grants, or perform
|
|
615
|
-
per-action confirmation.
|
|
616
|
-
- **BR-22:** The default BrowserControlPolicy permits all supported operations.
|
|
617
|
-
A Host may remove operations at launch; a running Agent cannot expand that
|
|
618
|
-
fixed policy through the browser tool protocol.
|
|
619
|
-
- **BR-23:** Every Workspace can inventory the same eligible user tab under a
|
|
620
|
-
distinct opaque ID. The single controlling-Lease rule in **BR-6** resolves
|
|
621
|
-
concurrent mutation.
|
|
622
|
-
- **BR-24:** Releasing a Lease or Workspace removes its user-target mappings and
|
|
623
|
-
invalidates their Observations without closing those tabs.
|
|
624
|
-
- **BR-25:** Eligible popups created from user tabs appear as ordinary user tabs.
|
|
625
|
-
Managed popup inheritance remains restricted to a complete verified opener
|
|
626
|
-
chain rooted in the same ManagedTabSet.
|
|
627
|
-
- **BR-26:** `tabs.list` returns eligible current and future tabs, tagging each
|
|
628
|
-
with origin and ManagedTabSet membership. Browser Pilot internal targets,
|
|
629
|
-
DevTools, extension-owned pages, and unsupported browser-internal targets
|
|
630
|
-
remain excluded. Excluded extension pages do not create or represent a
|
|
631
|
-
ProfileContext.
|
|
632
|
-
- **BR-27:** Bulk cleanup and `close --all` default to the ManagedTabSet. No
|
|
633
|
-
bulk operation closes user tabs outside it. Each user tab requires an
|
|
634
|
-
explicit target-specific close command.
|
|
635
|
-
- **BR-29:** Runtime Profile contexts are discovered from live target metadata,
|
|
636
|
-
not only `Target.getBrowserContexts`. Public ProfileContext IDs never expose
|
|
637
|
-
raw CDP identifiers and are invalid after browser reconnect.
|
|
638
|
-
- **BR-30:** Profile listing is passive. It cannot create, attach, activate, or
|
|
639
|
-
navigate a target solely to infer a Profile name. Identity requires the
|
|
640
|
-
explicit protocol 1.3 operation, exact Profile-path verification, bounded
|
|
641
|
-
metadata reads, and verified cleanup. Unverified names are omitted instead
|
|
642
|
-
of guessed.
|
|
643
|
-
- **BR-31:** New-target Profile routing follows explicit context, selected target,
|
|
644
|
-
Workspace selection, then single available context. Multiple unanchored
|
|
645
|
-
contexts fail with `profile_selection_required` before browser dispatch.
|
|
646
|
-
- **BR-32:** Every ManagedTabSet is bound to at most one ProfileContext. Managed
|
|
647
|
-
target creation verifies the returned context before public registration, and
|
|
648
|
-
fallback-created targets enter janitor ownership before they become
|
|
649
|
-
addressable. Fallback ownership requires the exact new URL marker plus either
|
|
650
|
-
the expected opener ID or exact readback of a per-command random window name;
|
|
651
|
-
Profile membership alone is never sufficient.
|
|
652
|
-
- **BR-33:** Switching to an existing target may update the owning Workspace's
|
|
653
|
-
selected ProfileContext, but never prevents inventory or control of targets
|
|
654
|
-
in other Profiles. No Profile selection is a permission grant.
|
|
655
|
-
|
|
656
|
-
## Command Reliability
|
|
657
|
-
|
|
658
|
-
Mutating commands follow this runtime state model:
|
|
80
|
+
## 5. Browser Scope and Authorization
|
|
659
81
|
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
\-> unknown_outcome
|
|
663
|
-
accepted -> cancelled
|
|
664
|
-
accepted -> expired
|
|
665
|
-
```
|
|
82
|
+
Browser Pilot exposes all eligible page tabs on the authorized browser
|
|
83
|
+
endpoint:
|
|
666
84
|
|
|
667
|
-
-
|
|
668
|
-
|
|
669
|
-
-
|
|
670
|
-
duplicate dispatched command returns its status and is not dispatched again.
|
|
671
|
-
- **BR-10:** If the Broker loses certainty after dispatch, it reports
|
|
672
|
-
`unknown_outcome`. Mutating commands are never automatically replayed.
|
|
673
|
-
- **BR-11:** Cancellation before dispatch prevents execution. Cancellation
|
|
674
|
-
after dispatch is best-effort and must not claim rollback.
|
|
675
|
-
- **BR-12:** Browser disconnect and explicit reconnect change the BrowserInstance
|
|
676
|
-
connection generation and invalidate sessions and observations. The generation
|
|
677
|
-
advances on successful restoration, not merely on an attempt. Discovery and
|
|
678
|
-
timers never initiate a browser connection.
|
|
679
|
-
|
|
680
|
-
Commands capture that generation when accepted. The Broker verifies it again
|
|
681
|
-
before browser dispatch and after the executor returns. A queued stale command
|
|
682
|
-
never reaches the browser. If a mutating command was already dispatched when the
|
|
683
|
-
generation changed, its terminal state is `unknown_outcome`; a stale read fails
|
|
684
|
-
with retryable `browser_disconnected` rather than returning old data.
|
|
685
|
-
|
|
686
|
-
The dispatch transition occurs immediately before the first browser mutation,
|
|
687
|
-
after argument, ref, target, and continuity preflight. Composite-action guards
|
|
688
|
-
preserve their structured interruption context when the terminal state is
|
|
689
|
-
`unknown_outcome`; callers must obtain fresh target/frame/Observation state
|
|
690
|
-
before deciding what remains to be done.
|
|
691
|
-
|
|
692
|
-
Normal bridge calls preserve dispatch order. `commands/get`, `commands/cancel`,
|
|
693
|
-
and dialog list/respond calls may bypass a pending `tools/call`; JSON-RPC clients
|
|
694
|
-
therefore correlate responses by ID rather than relying on response order. The
|
|
695
|
-
dialog exception prevents a paused browser command from blocking the explicit
|
|
696
|
-
response needed to release its JavaScript dialog.
|
|
697
|
-
|
|
698
|
-
Workspaces, Leases, target mappings, refs, command status, idempotency records,
|
|
699
|
-
and bounded event journals live in Broker memory by default. Broker restart
|
|
700
|
-
invalidates them; clients initialize again, list tabs again, and observe again.
|
|
701
|
-
Only installation identity, browser preference, Broker locator/version data,
|
|
702
|
-
and explicit user configuration may persist. DOM snapshots, refs, cookies,
|
|
703
|
-
passwords, network bodies, and transient control state never persist.
|
|
704
|
-
|
|
705
|
-
Broker-created targets cross one additional private ownership boundary. A
|
|
706
|
-
janitor child process is the sole owner of the root CDP WebSocket and proxies
|
|
707
|
-
all daemon CDP requests, results, and events over private IPC. The same process
|
|
708
|
-
creates and tracks only managed targets plus popup descendants with a verified
|
|
709
|
-
managed opener chain. Its parent pipe is the liveness signal: daemon EOF,
|
|
710
|
-
including `SIGKILL`, causes bounded descendant-first target cleanup. User
|
|
711
|
-
targets are never adopted merely because they share a browser window. Browser
|
|
712
|
-
disconnect clears that transient ownership before a new connection generation
|
|
713
|
-
is exposed. No janitor method or raw target ID is part of the CLI or stdio
|
|
714
|
-
protocol, and no ownership record is written to disk.
|
|
715
|
-
|
|
716
|
-
## Observation and Ref Semantics
|
|
717
|
-
|
|
718
|
-
Observation v1 has these required public fields: `workspaceId`, `leaseId`,
|
|
719
|
-
`targetId`, `url`, `observationId`, `title`, `elements`, `truncated`, and
|
|
720
|
-
`truncationReasons`. Each element contains only `ref`, `role`, `name`, and the
|
|
721
|
-
optional `value` and `checked` fields. Built-in Observation-producing tools
|
|
722
|
-
also return `hints`; action tools additionally return discriminated `evidence`.
|
|
723
|
-
The latter two fields remain optional in the schema so older compatible
|
|
724
|
-
executors stay valid. Raw nodes, selectors, bounds, CDP identifiers, and the
|
|
725
|
-
internal identity below never appear in an Observation result.
|
|
726
|
-
|
|
727
|
-
The frozen v1 limits are 50 elements by default and 10,000 at most; 4,096
|
|
728
|
-
string code units for title and element name; 16,384 for URL; 65,536 for one
|
|
729
|
-
element value; 1,000,000 across collected Observation text; AX depth 128; and
|
|
730
|
-
2 MiB of UTF-8 serialized Observation data. Broker storage retains at most
|
|
731
|
-
2,048 Observations for five minutes each. Hitting any collection limit sets
|
|
732
|
-
`truncated: true` and includes one or more canonical reasons in deterministic
|
|
733
|
-
order: `element_limit`, `text_limit`, `depth_limit`, `byte_limit`.
|
|
734
|
-
|
|
735
|
-
The public identity of an element is:
|
|
85
|
+
- managed tabs created by Browser Pilot;
|
|
86
|
+
- eligible popup descendants of managed tabs;
|
|
87
|
+
- eligible tabs opened by the user in every live profile.
|
|
736
88
|
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
```
|
|
89
|
+
Browser-internal pages, extension-owned targets, and targets that cannot be
|
|
90
|
+
controlled safely are excluded.
|
|
740
91
|
|
|
741
|
-
|
|
92
|
+
Chrome tab-group membership does not affect tab eligibility. Browser Pilot can
|
|
93
|
+
control an eligible tab in an expanded or collapsed group but, without an
|
|
94
|
+
extension, cannot inspect, create, rename, move, collapse, or delete groups.
|
|
742
95
|
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
browserConnectionGeneration
|
|
746
|
-
targetId
|
|
747
|
-
CDPSessionId
|
|
748
|
-
frameId
|
|
749
|
-
loaderId
|
|
750
|
-
backendNodeId
|
|
751
|
-
documentGeneration
|
|
752
|
-
```
|
|
96
|
+
Browser Pilot itself does not ask for task-intent approval. The authorization
|
|
97
|
+
boundaries are:
|
|
753
98
|
|
|
754
|
-
The
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
Same-context failures expose one of these stable invalidation reasons when
|
|
758
|
-
known: `navigation`, `loader_replaced`, `document_replaced`, `frame_changed`,
|
|
759
|
-
`frame_detached`, `session_replaced`, `target_detached`,
|
|
760
|
-
`browser_reconnected`, `target_ineligible`, `target_closed`,
|
|
761
|
-
`control_released`, or `expired`.
|
|
762
|
-
Ownership mismatches remain an undifferentiated `stale_ref` so they do not
|
|
763
|
-
disclose another Workspace's state.
|
|
764
|
-
|
|
765
|
-
Adding `document_replaced` as an optional `stale_ref.context.reason` value is
|
|
766
|
-
additive under protocol 1.0 and 1.1: clients branch on the stable error code,
|
|
767
|
-
handle known reasons when useful, and treat unknown reasons as a request to
|
|
768
|
-
rebuild current Observation state. It adds no operation or capability.
|
|
769
|
-
|
|
770
|
-
- **BR-13:** Cross-Workspace, cross-target, cross-observation, and expired refs
|
|
771
|
-
return `stale_ref`; they never fall back to another target.
|
|
772
|
-
- **BR-14:** Navigation, loader or Document replacement, target detach, frame
|
|
773
|
-
detach, CDP session replacement, and browser reconnect hard-invalidate
|
|
774
|
-
affected refs.
|
|
775
|
-
- **BR-15:** Same-document DOM changes require live node revalidation. They do
|
|
776
|
-
not force blanket invalidation when the node remains resolvable and valid.
|
|
777
|
-
- **BR-16:** Observations are immutable and bounded by element count, text size,
|
|
778
|
-
depth, and byte limits, with explicit truncation metadata.
|
|
779
|
-
- **BR-28:** Agent hints are advisory, deterministic, bounded, and based only on
|
|
780
|
-
browser-verifiable signals. They never contain field values, credentials,
|
|
781
|
-
private file paths, raw CDP identities, or framework-specific prompt state.
|
|
782
|
-
|
|
783
|
-
## Events
|
|
784
|
-
|
|
785
|
-
The event taxonomy includes navigation, document changed, target attached,
|
|
786
|
-
target detached, control acquired/released, popup, dialog, download, connection
|
|
787
|
-
lost, connection restored, network request, network response, command status,
|
|
788
|
-
observation invalidated, Lease expiry, and watchdog signals for stalled
|
|
789
|
-
navigation, selected-frame detach, unhandled dialogs, and repeated observable
|
|
790
|
-
no-progress actions.
|
|
791
|
-
|
|
792
|
-
Relevant event payloads carry the same `hints` representation: main-document
|
|
793
|
-
403/429 responses use `access_blocked`; downloads use `download` with the public
|
|
794
|
-
Artifact ID only after completion; and the threshold no-progress event uses
|
|
795
|
-
`repeated_action`. Subresource failures do not produce a page-blocked hint.
|
|
796
|
-
|
|
797
|
-
Every event contains an event ID, monotonic Workspace sequence, timestamp,
|
|
798
|
-
Workspace ID, relevant Lease and target IDs, type, payload version, and
|
|
799
|
-
sensitivity. `workspaces/create|get` returns an explicit current `eventCursor`.
|
|
800
|
-
Consumers poll from their last fully processed cursor; a poll returns ordered
|
|
801
|
-
events, `nextCursor`, and `hasMore`. The Broker may compact old events and must
|
|
802
|
-
return `cursor_expired` with retained cursor bounds rather than silently skip
|
|
803
|
-
them. A consumer that loses its cursor rebuilds tab and Observation state, then
|
|
804
|
-
uses `workspaces/get` to establish a new baseline.
|
|
805
|
-
|
|
806
|
-
JavaScript dialogs enter explicit pending state and emit opened/closed events.
|
|
807
|
-
They remain paused until the user or an Agent issues `browser.dialogs.respond`;
|
|
808
|
-
Browser Pilot never auto-accepts them.
|
|
809
|
-
|
|
810
|
-
Watchdogs report browser state; they do not make task decisions. A navigation
|
|
811
|
-
that never becomes interactive emits `watchdog.navigation_stalled` and leaves
|
|
812
|
-
the dispatched command as `unknown_outcome`. Detaching the selected frame emits
|
|
813
|
-
`watchdog.frame_detached`, clears the frame selection, and invalidates its
|
|
814
|
-
Observation. A dialog still pending after the bounded threshold emits one
|
|
815
|
-
`watchdog.dialog_unhandled` event but remains pending. Three consecutive actions
|
|
816
|
-
with browser-observable mismatch or no effect emit one `watchdog.no_progress`
|
|
817
|
-
event per streak, scoped by Lease and target. Unobservable coordinate/canvas
|
|
818
|
-
actions do not increment the streak. Navigation, frame changes, verified
|
|
819
|
-
progress, session replacement, Lease release, and Workspace cleanup reset the
|
|
820
|
-
applicable transient state.
|
|
821
|
-
|
|
822
|
-
Watchdog payloads contain only bounded recovery metadata and opaque public IDs;
|
|
823
|
-
they never contain typed values, passwords, refs, raw CDP target/session/frame
|
|
824
|
-
IDs, or dialog prompt responses. Consumers inspect fresh browser state before
|
|
825
|
-
deciding whether to continue, retry, or ask the user.
|
|
826
|
-
|
|
827
|
-
## Artifacts
|
|
828
|
-
|
|
829
|
-
Artifact descriptors contain an opaque ID, kind, MIME type, byte size, original
|
|
830
|
-
filename or dimensions when applicable, sensitivity, creation time, expiry, and
|
|
831
|
-
optional preview relationship. They do not expose internal storage paths as the
|
|
832
|
-
primary identity.
|
|
833
|
-
|
|
834
|
-
- **BR-17:** Broker storage directories are mode `0700`; files are mode `0600`
|
|
835
|
-
on platforms that support POSIX permissions.
|
|
836
|
-
- **BR-18:** Artifacts have per-item, per-Workspace, and global capacity limits,
|
|
837
|
-
TTL cleanup, retain/release, and explicit export to a client-owned path.
|
|
838
|
-
- **BR-19:** Binary bytes are not written as base64 to the stdio stream. The
|
|
839
|
-
local adapter reads an authorized Artifact or asks the CLI to export it, then
|
|
840
|
-
converts it to the Agent runtime's image/file content type.
|
|
841
|
-
- **BR-20:** Large or full-page screenshots return a model-sized preview plus
|
|
842
|
-
an optional original Artifact.
|
|
843
|
-
|
|
844
|
-
An adapter authorizes a local upload by calling `artifacts/import` with an
|
|
845
|
-
absolute path, Workspace, and active Lease. The Broker copies the file into
|
|
846
|
-
protected storage as a `user_file` / `upload_input` Artifact while preserving
|
|
847
|
-
its basename. `browser.upload` accepts only that Artifact kind and never treats
|
|
848
|
-
a screenshot, PDF, download, or arbitrary Broker-internal path as upload
|
|
849
|
-
authorization.
|
|
850
|
-
|
|
851
|
-
Downloads are event-produced Artifacts rather than a separate path-returning
|
|
852
|
-
tool. CDP download behavior is browser-context scoped even when invoked through
|
|
853
|
-
an attached target session, so per-session download directories cannot provide
|
|
854
|
-
isolation. For each browser connection generation and Chrome Profile context,
|
|
855
|
-
the Broker enables `Browser.downloadProgress` once with
|
|
856
|
-
`Browser.setDownloadBehavior({ behavior: "default", eventsEnabled: true })`.
|
|
857
|
-
The `default` behavior preserves Chrome's configured download directory and
|
|
858
|
-
does not redirect user downloads.
|
|
859
|
-
|
|
860
|
-
`Page.downloadWillBegin` on an attached controlled target binds the private CDP
|
|
861
|
-
GUID to that target's Workspace and Lease. The browser-level completed event
|
|
862
|
-
supplies the absolute source `filePath`; the Broker accepts it only for a GUID
|
|
863
|
-
already bound by the Page event, validates a regular file, and copies it into a
|
|
864
|
-
Workspace-owned `download` Artifact before publishing completion. Browser-level
|
|
865
|
-
events without a controlled Page owner are ignored. GUIDs and source paths never
|
|
866
|
-
appear in protocol output.
|
|
867
|
-
|
|
868
|
-
The Browser Pilot copy is subject to tracking, item, Workspace, and global
|
|
869
|
-
Artifact bounds. Exceeding a bound rejects only Artifact creation: it must not
|
|
870
|
-
cancel, move, truncate, or delete Chrome's original download. Lease, session,
|
|
871
|
-
Workspace, Artifact, and Broker cleanup stop tracking and delete only protected
|
|
872
|
-
Artifact copies. The original file in the user's Chrome download directory is
|
|
873
|
-
always left intact. If browser-level download events are unavailable, Browser
|
|
874
|
-
Pilot reports capture as unavailable without changing Chrome's download
|
|
875
|
-
behavior.
|
|
876
|
-
|
|
877
|
-
## Browser Discovery and Setup
|
|
878
|
-
|
|
879
|
-
Discovery returns every supported local browser candidate with product,
|
|
880
|
-
channel, user-data root, process state, remote-debugging state, authorization state,
|
|
881
|
-
and structured remediation. It does not silently select the first filesystem
|
|
882
|
-
match when several viable instances exist.
|
|
883
|
-
|
|
884
|
-
After connection, Profile discovery separately returns every live regular
|
|
885
|
-
Profile context on that endpoint. Browser candidates and Profile contexts are
|
|
886
|
-
different layers: selecting Chrome Stable does not silently select one of its
|
|
887
|
-
simultaneously open Profiles.
|
|
888
|
-
|
|
889
|
-
The client may select an instance explicitly or use a deterministic persisted
|
|
890
|
-
preference. Setup that requires Chrome UI remains a user-visible action. Browser
|
|
891
|
-
Pilot cannot bypass Chrome authorization and must not simulate consent.
|
|
892
|
-
|
|
893
|
-
Browser Pilot does not add an access-approval page. Products that want an
|
|
894
|
-
approval UX own it in the Agent host before exposing the tool. Browser Pilot
|
|
895
|
-
still excludes its own setup/status pages and other internal targets from
|
|
896
|
-
Agent inventory; the same-user CDP limitation in **CON-7** remains.
|
|
897
|
-
|
|
898
|
-
## Security and Sensitive Data
|
|
899
|
-
|
|
900
|
-
- Password inputs, cookies, auth credentials, network bodies, uploads,
|
|
901
|
-
downloads, page captures, and selected page text carry sensitivity metadata.
|
|
902
|
-
- Tool schemas mark selected text and page/element values as `browser_data`,
|
|
903
|
-
with `credential` added where a field may carry a password, cookie, header,
|
|
904
|
-
request body, response body, prompt response, or arbitrary eval value.
|
|
905
|
-
- Artifact descriptors and BrowserEvents carry runtime sensitivity because
|
|
906
|
-
their classification depends on the produced object or event. Their runtime
|
|
907
|
-
value takes precedence over a tool's possible schema classifications.
|
|
908
|
-
- Secrets are accepted through protected stdin or structured machine input, not
|
|
909
|
-
command-line arguments in recommended workflows.
|
|
910
|
-
- Audit records store metadata and hashes where possible, not secret values.
|
|
911
|
-
- Public errors do not include cookies, credentials, network bodies, DOM dumps,
|
|
912
|
-
or raw CDP payloads.
|
|
913
|
-
- The Broker socket/pipe, configuration, and Artifact storage are restricted to
|
|
914
|
-
the OS user.
|
|
915
|
-
- Browser Pilot documents that another same-user process may bypass it and use
|
|
916
|
-
Chrome's debugging endpoint directly.
|
|
917
|
-
|
|
918
|
-
## Stable Error Model
|
|
919
|
-
|
|
920
|
-
Errors contain a stable code, human message, retryability, affected object IDs,
|
|
921
|
-
and optional structured remediation. Required codes include:
|
|
99
|
+
1. The user enables Chrome remote debugging.
|
|
100
|
+
2. Chrome presents its own connection authorization dialog when required.
|
|
101
|
+
3. The Agent host decides whether the Agent may invoke local commands.
|
|
922
102
|
|
|
923
|
-
|
|
924
|
-
|
|
925
|
-
browser_not_found browser_not_authorized browser_disconnected
|
|
926
|
-
broker_in_use
|
|
927
|
-
profile_selection_required profile_context_stale profile_context_unavailable
|
|
928
|
-
workspace_not_found lease_expired target_not_owned
|
|
929
|
-
target_busy stale_ref command_cancelled
|
|
930
|
-
action_not_verified command_expired unknown_outcome
|
|
931
|
-
artifact_not_found
|
|
932
|
-
artifact_expired cursor_expired result_too_large
|
|
933
|
-
invalid_argument internal_error
|
|
934
|
-
```
|
|
103
|
+
A host may restrict operations through its own policy. Browser Pilot's default
|
|
104
|
+
surface remains complete and does not implement an approval state machine.
|
|
935
105
|
|
|
936
|
-
|
|
937
|
-
fields, never on message text.
|
|
938
|
-
|
|
939
|
-
## Compatibility and Versioning
|
|
940
|
-
|
|
941
|
-
- Protocol versions are independent of executable versions.
|
|
942
|
-
- Compatible embedded clients reuse the running Broker even when their host or
|
|
943
|
-
bundled executable versions differ. The one-shot compatibility CLI requires
|
|
944
|
-
the running Broker's exact executable version because its external JSON
|
|
945
|
-
contract is versioned separately from the bridge protocol.
|
|
946
|
-
- An incompatible executable never replaces a live Broker. A host either uses
|
|
947
|
-
a compatible executable or deliberately supplies a distinct
|
|
948
|
-
`BROWSER_PILOT_HOME`; Browser Pilot never creates a hidden version namespace.
|
|
949
|
-
- Broker shutdown is compare-and-stop: it requires the current Broker process
|
|
950
|
-
identity and matching executable installation identity, and refuses while
|
|
951
|
-
any embedded Connection or active Lease remains live.
|
|
952
|
-
- Additive tool and field changes use capability negotiation and schema
|
|
953
|
-
evolution. Breaking semantics require a protocol-major change.
|
|
954
|
-
- Unknown response fields are ignored; unknown request fields are rejected only
|
|
955
|
-
when they change safety or semantics.
|
|
956
|
-
- Optional advisory output such as `hints` does not require a protocol-minor
|
|
957
|
-
version or capability when it adds no operation and only derives bounded data
|
|
958
|
-
already covered by an existing granted capability.
|
|
959
|
-
- The current CLI JSON shapes remain supported until a documented major CLI
|
|
960
|
-
release and migration period.
|
|
961
|
-
- A conformance suite validates any embedded adapter against the same executable
|
|
962
|
-
used by direct CLI users.
|
|
963
|
-
|
|
964
|
-
## Functional Requirements and Acceptance
|
|
965
|
-
|
|
966
|
-
- **FR-1 Universal installation:** An Agent can install Browser Pilot globally
|
|
967
|
-
or locally and use existing one-shot commands.
|
|
968
|
-
- **AC-1:** Existing documented CLI workflows and fixture tests pass after
|
|
969
|
-
Broker migration without requiring an SDK, MCP server, or extension.
|
|
970
|
-
- **FR-2 Embedded integration:** A product can bundle the official executable,
|
|
971
|
-
initialize `bridge --stdio`, discover tools, execute them, receive events,
|
|
972
|
-
and release resources.
|
|
973
|
-
- **AC-2:** A reference adapter integrates using only executable invocation,
|
|
974
|
-
the published protocol, and Artifact files; it imports no Browser Pilot
|
|
975
|
-
source module.
|
|
976
|
-
- **FR-3 Multi-client isolation:** Concurrent clients have isolated control and
|
|
977
|
-
transient state.
|
|
978
|
-
- **AC-3:** Two clients can operate separate Pilot windows concurrently, and
|
|
979
|
-
refs, selected targets, frames, auth, rules, and close operations cannot cross
|
|
980
|
-
Workspace boundaries.
|
|
981
|
-
- **FR-4 User-browser control:** An Agent can list and control all eligible
|
|
982
|
-
current and future tabs in the selected BrowserInstance, including tabs the
|
|
983
|
-
user opened before invoking it.
|
|
984
|
-
- **AC-4:** No access-request or grant step is required. Inventory combines
|
|
985
|
-
ManagedTabSets and user tabs behind Workspace-scoped opaque IDs. A Host may
|
|
986
|
-
remove operations at launch. Concurrent control reports `target_busy`,
|
|
987
|
-
release leaves user tabs open, and bulk cleanup closes only managed tabs.
|
|
988
|
-
- **FR-5 Crash behavior:** Client and Broker failures have deterministic
|
|
989
|
-
outcomes.
|
|
990
|
-
- **AC-5:** Expired Leases are reclaimed; commands lost after dispatch report
|
|
991
|
-
`unknown_outcome`; reconnect invalidates prior observations.
|
|
992
|
-
- **FR-6 Media delivery:** Agent products can deliver Browser Pilot screenshots
|
|
993
|
-
to multimodal models.
|
|
994
|
-
- **AC-6:** Capture returns a protected Artifact and preview metadata; a
|
|
995
|
-
reference adapter converts it to native image content without base64 on
|
|
996
|
-
stdout.
|
|
997
|
-
- **FR-7 Version safety:** User-installed and product-bundled clients coexist.
|
|
998
|
-
- **AC-7:** Compatible clients reuse the Broker; incompatible clients receive
|
|
999
|
-
`protocol_incompatible` or use a deliberately isolated Broker without
|
|
1000
|
-
terminating live clients.
|
|
1001
|
-
- **FR-8 Browser reliability:** Browser capability improvements do not alter the
|
|
1002
|
-
public lifecycle and isolation contract.
|
|
1003
|
-
- **AC-8:** Observation and action conformance tests pass against the public
|
|
1004
|
-
tool interface, including frame, navigation, stale-ref, obstruction, and
|
|
1005
|
-
input readback cases.
|
|
1006
|
-
- **FR-9 Multi-Profile routing:** One endpoint can expose all eligible tabs from
|
|
1007
|
-
multiple live Chrome Profiles without ambiguous new-target placement.
|
|
1008
|
-
- **AC-9:** Existing tabs remain directly controllable; every tab result has a
|
|
1009
|
-
connection-scoped ProfileContext ID; ambiguous open returns
|
|
1010
|
-
`profile_selection_required`; explicit or inferred routing is verified; and
|
|
1011
|
-
reconnect requires relisting rather than reusing stale Profile IDs.
|
|
1012
|
-
|
|
1013
|
-
## Non-Functional Requirements
|
|
1014
|
-
|
|
1015
|
-
- **NFR-1:** No protocol message or normal tool result exceeds negotiated size
|
|
1016
|
-
limits; truncation is explicit.
|
|
1017
|
-
- **NFR-2:** Broker restart invalidates transient Workspaces, Leases, target
|
|
1018
|
-
assignments, refs, and commands. Clients rebuild state explicitly without
|
|
1019
|
-
guessing or replaying mutations.
|
|
1020
|
-
- **NFR-3:** Every command and security-sensitive lifecycle transition emits an
|
|
1021
|
-
auditable event without storing secret payloads.
|
|
1022
|
-
- **NFR-4:** Target command ordering is deterministic under concurrent clients.
|
|
1023
|
-
- **NFR-5:** Resource cleanup is idempotent and bounded after normal release,
|
|
1024
|
-
client crash, browser exit, and Broker restart.
|
|
1025
|
-
|
|
1026
|
-
## Non-Goals
|
|
1027
|
-
|
|
1028
|
-
- Hosting browsers in the cloud or providing captcha-solving infrastructure.
|
|
1029
|
-
- Controlling Firefox or WebKit in the current architecture.
|
|
1030
|
-
- Providing OS-level isolation between processes owned by the same user.
|
|
1031
|
-
- Defining whether an Agent's requested browser action matches user intent;
|
|
1032
|
-
Agent hosts own semantic approval and confirmation policy.
|
|
1033
|
-
- Modeling an Agent's conversation, run, task, worker, or model provider.
|
|
1034
|
-
- Replacing URL Preview features inside embedding products.
|
|
1035
|
-
- Making Browser Pilot an MCP server or publishing language SDKs.
|
|
1036
|
-
|
|
1037
|
-
## Verification Strategy
|
|
1038
|
-
|
|
1039
|
-
1. Protocol unit tests validate framing, negotiation, errors, schemas,
|
|
1040
|
-
idempotency, cancellation, and cursor behavior.
|
|
1041
|
-
2. Broker integration tests create concurrent fake clients and assert complete
|
|
1042
|
-
tab inventory, target assignment, Lease cleanup, crash recovery, and version
|
|
1043
|
-
coexistence.
|
|
1044
|
-
3. Existing Playwright fixtures remain regression tests for human CLI behavior.
|
|
1045
|
-
4. Capability tests cover DOM/AX fusion, frames/OOPIF, stale refs, obstruction,
|
|
1046
|
-
action verification, input readback, and document changes.
|
|
1047
|
-
5. Adapter conformance tests run against a language-neutral reference adapter
|
|
1048
|
-
and at least two example Agent runtimes. No example-specific behavior enters
|
|
1049
|
-
Broker production code.
|
|
1050
|
-
6. Browser-control tests prove that all eligible existing and future tabs are
|
|
1051
|
-
included without grants, launch-time denied operations fail before CDP
|
|
1052
|
-
dispatch, opaque IDs cannot cross Workspaces, and bulk cleanup never closes
|
|
1053
|
-
user tabs.
|
|
1054
|
-
7. Distribution tests pack and install the npm package through global and local
|
|
1055
|
-
layouts, launch local npx and product-owned absolute paths, and exercise
|
|
1056
|
-
version output, bridge initialization, private Broker startup, and cleanup.
|
|
1057
|
-
Native verification additionally checks manifest hashes, signature state,
|
|
1058
|
-
private child roles, and archive checksums on each target platform.
|
|
1059
|
-
8. Multi-Profile tests cover passive discovery, explicit and inferred
|
|
1060
|
-
selection, multiple ManagedTabSets, direct-create fallback, janitor adoption,
|
|
1061
|
-
concurrent Workspace routing, reconnect invalidation, and protocol 1.1
|
|
1062
|
-
ambiguous-open failure. Real Chrome acceptance verifies at least two live
|
|
1063
|
-
Profiles on one endpoint without reconnecting or requesting authorization
|
|
1064
|
-
again.
|
|
1065
|
-
|
|
1066
|
-
## Implementation Boundary
|
|
1067
|
-
|
|
1068
|
-
The target internal boundary is:
|
|
106
|
+
## 6. Browser Discovery and Connection
|
|
1069
107
|
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
|
|
1073
|
-
|
|
1074
|
-
|
|
1075
|
-
|
|
108
|
+
Discovery returns structured candidates with:
|
|
109
|
+
|
|
110
|
+
- stable browser ID;
|
|
111
|
+
- product and channel;
|
|
112
|
+
- user-data root;
|
|
113
|
+
- process state;
|
|
114
|
+
- remote-debugging state;
|
|
115
|
+
- authorization state;
|
|
116
|
+
- aggregate setup state;
|
|
117
|
+
- structured remediation.
|
|
118
|
+
|
|
119
|
+
Discovery is passive. It must not open a TCP or WebSocket connection and must
|
|
120
|
+
not display Chrome's Allow dialog.
|
|
121
|
+
|
|
122
|
+
Only an explicit browser connection command requests authorization. Concurrent
|
|
123
|
+
clients share one in-flight connection attempt. A failed or dropped connection
|
|
124
|
+
is never retried by a periodic timer. The next explicit command may request
|
|
125
|
+
reconnection.
|
|
126
|
+
|
|
127
|
+
One supervised browser-level CDP WebSocket is shared per endpoint. It must not
|
|
128
|
+
be multiplied per CLI process, Agent, profile, or tab.
|
|
129
|
+
|
|
130
|
+
## 7. Profiles
|
|
131
|
+
|
|
132
|
+
One browser endpoint may expose multiple live Chrome profile contexts. Browser
|
|
133
|
+
Pilot must:
|
|
134
|
+
|
|
135
|
+
- inventory contexts passively from live target state;
|
|
136
|
+
- return representative eligible tabs;
|
|
137
|
+
- use connection-generation-scoped opaque profile IDs;
|
|
138
|
+
- route new managed tabs only after an unambiguous selection;
|
|
139
|
+
- preserve access to existing eligible tabs regardless of profile selection;
|
|
140
|
+
- invalidate profile IDs on browser reconnect.
|
|
141
|
+
|
|
142
|
+
Account-aware identity is an explicit visible operation. Browser Pilot may
|
|
143
|
+
open one temporary `chrome://version` page in each unidentified context, match
|
|
144
|
+
the exact profile path to bounded Local State metadata, then close the page.
|
|
145
|
+
It must not infer account identity from profile directory order or names.
|
|
146
|
+
|
|
147
|
+
If identity is unavailable or ambiguous, the result uses a neutral label and
|
|
148
|
+
representative tabs. The Agent must not silently select the first profile.
|
|
149
|
+
|
|
150
|
+
## 8. Agent State Model
|
|
151
|
+
|
|
152
|
+
The public concept is an Agent namespace selected by
|
|
153
|
+
`BROWSER_PILOT_CLIENT_KEY` or `--client-key`.
|
|
154
|
+
|
|
155
|
+
Internally, the Broker maintains a principal, logical workspace, renewable
|
|
156
|
+
control lease, target mappings, selected profile/tab/frame, observations,
|
|
157
|
+
auth/network configuration, downloads, artifacts, commands, and bounded event
|
|
158
|
+
journal. These internal records are not exposed as a host protocol.
|
|
159
|
+
|
|
160
|
+
Requirements:
|
|
161
|
+
|
|
162
|
+
- The same stable key resumes one Agent's state across CLI processes.
|
|
163
|
+
- Different keys cannot inspect or mutate one another's state.
|
|
164
|
+
- A key must not be generated per command.
|
|
165
|
+
- State lives in Broker memory, not global refs or active-tab files.
|
|
166
|
+
- Browser reconnect invalidates all old browser-generation-scoped addresses.
|
|
167
|
+
- Expiry and explicit disconnect reclaim transient state and managed targets.
|
|
168
|
+
- User-opened tabs remain open during cleanup.
|
|
169
|
+
|
|
170
|
+
Compatible CLI versions may establish separate internal client sessions while
|
|
171
|
+
sharing the same Agent principal and keyed state. Compatibility is determined
|
|
172
|
+
by protocol negotiation, not executable path equality.
|
|
173
|
+
|
|
174
|
+
## 9. Target Model and Concurrency
|
|
175
|
+
|
|
176
|
+
Every Agent receives opaque controlled-target addresses. Raw CDP target IDs and
|
|
177
|
+
session IDs are internal.
|
|
178
|
+
|
|
179
|
+
The inventory records origin as:
|
|
180
|
+
|
|
181
|
+
- `managed`;
|
|
182
|
+
- `managed_popup`;
|
|
183
|
+
- `user_tab`.
|
|
184
|
+
|
|
185
|
+
Requirements:
|
|
186
|
+
|
|
187
|
+
- The same physical user tab may appear in several Agent inventories.
|
|
188
|
+
- At most one Agent may control a physical tab at a time.
|
|
189
|
+
- A busy target returns stable `target_busy`; another Agent cannot steal it.
|
|
190
|
+
- Commands for the same physical target execute serially.
|
|
191
|
+
- Commands for independent targets may run concurrently.
|
|
192
|
+
- Explicit release removes the controlling session before another Agent can
|
|
193
|
+
acquire the target.
|
|
194
|
+
- A tab becoming ineligible invalidates its mapping without closing the tab.
|
|
195
|
+
- Bulk close operates only on managed tabs.
|
|
196
|
+
- Explicit close may close a selected user tab.
|
|
197
|
+
|
|
198
|
+
Managed popup adoption requires a complete owned opener chain. Browser Pilot
|
|
199
|
+
must fail closed when popup ownership cannot be proven.
|
|
200
|
+
|
|
201
|
+
## 10. Observation and Refs
|
|
202
|
+
|
|
203
|
+
An observation fuses bounded accessibility semantics with DOM layout and state.
|
|
204
|
+
It includes:
|
|
205
|
+
|
|
206
|
+
- numbered interactive refs;
|
|
207
|
+
- role, accessible name, bounded value, and checked state;
|
|
208
|
+
- visibility, enabled/editable state, and geometry;
|
|
209
|
+
- open Shadow DOM;
|
|
210
|
+
- selected frame context;
|
|
211
|
+
- readable text and page geometry where requested;
|
|
212
|
+
- truncation metadata.
|
|
213
|
+
|
|
214
|
+
Each ref is scoped internally by Agent state, browser generation, target, CDP
|
|
215
|
+
session, frame, document/loader identity, observation generation, and backing
|
|
216
|
+
node identity.
|
|
217
|
+
|
|
218
|
+
Refs become stale after navigation, document replacement, semantic mutation,
|
|
219
|
+
target detach, frame change, reconnect, state release, or a newer invalidating
|
|
220
|
+
observation. A stale ref returns `stale_ref`; it must never be silently rebound
|
|
221
|
+
to another element.
|
|
222
|
+
|
|
223
|
+
Observation output is bounded by element count, field sizes, tree depth, text
|
|
224
|
+
length, and serialized byte budget. Truncation is explicit.
|
|
225
|
+
|
|
226
|
+
## 11. Actions and Verification
|
|
227
|
+
|
|
228
|
+
Browser Pilot supports semantic click, type, press, keyboard input, upload,
|
|
229
|
+
scroll, and dropdown selection, plus coordinate input for visual surfaces.
|
|
230
|
+
|
|
231
|
+
Before dispatch, an element action must validate:
|
|
232
|
+
|
|
233
|
+
- current ref identity;
|
|
234
|
+
- target and frame continuity;
|
|
235
|
+
- enabled/editable state;
|
|
236
|
+
- live geometry;
|
|
237
|
+
- hit point and obstruction;
|
|
238
|
+
- browser connection generation.
|
|
239
|
+
|
|
240
|
+
After dispatch, Browser Pilot returns fresh page state and typed evidence. It
|
|
241
|
+
checks applicable effects such as:
|
|
242
|
+
|
|
243
|
+
- focus changes;
|
|
244
|
+
- checked/selected/pressed/expanded state;
|
|
245
|
+
- input or contenteditable value length and exact readback;
|
|
246
|
+
- navigation or document replacement;
|
|
247
|
+
- dialog or popup creation;
|
|
248
|
+
- upload file count and filename;
|
|
249
|
+
- scroll position;
|
|
250
|
+
- dropdown selection.
|
|
251
|
+
|
|
252
|
+
A mismatch returns a typed verification result or stable error. Sensitive
|
|
253
|
+
input evidence records only bounded metadata and never echoes password content.
|
|
254
|
+
|
|
255
|
+
If the page changes during a compound action, remaining steps are cancelled.
|
|
256
|
+
For example, type-with-submit must not press Enter after the input document is
|
|
257
|
+
replaced.
|
|
258
|
+
|
|
259
|
+
## 12. Page Representations
|
|
260
|
+
|
|
261
|
+
The Agent can choose the smallest representation needed:
|
|
262
|
+
|
|
263
|
+
- snapshot for semantic controls;
|
|
264
|
+
- readable text for articles, lists, and results;
|
|
265
|
+
- targeted text search for a phrase and nearby context;
|
|
266
|
+
- bounded DOM metadata for selectors and attributes;
|
|
267
|
+
- page and scroll geometry for long documents;
|
|
268
|
+
- annotated screenshots for visual position and overlap;
|
|
269
|
+
- JavaScript evaluation only as an escape hatch.
|
|
270
|
+
|
|
271
|
+
This adaptive model incorporates the strongest relevant browser-use ideas
|
|
272
|
+
without adopting browser-use's Agent runtime or Python API.
|
|
273
|
+
|
|
274
|
+
## 13. Async Operations and Command Recovery
|
|
275
|
+
|
|
276
|
+
The CLI provides browser-visible waits for URL, text, selector, dialog,
|
|
277
|
+
download, and popup conditions. Waiting uses a caller-defined deadline and
|
|
278
|
+
bounded polling interval. Timeout returns stable `wait_timeout` and does not
|
|
279
|
+
assert anything about an underlying mutation.
|
|
280
|
+
|
|
281
|
+
Every browser command has an internal command record with status:
|
|
282
|
+
|
|
283
|
+
- `accepted`;
|
|
284
|
+
- `dispatched`;
|
|
285
|
+
- `completed`;
|
|
286
|
+
- `unknown_outcome`;
|
|
287
|
+
- `cancelled`;
|
|
288
|
+
- `expired`.
|
|
289
|
+
|
|
290
|
+
The CLI exposes bounded command list, lookup, and cancellation commands.
|
|
291
|
+
`--request-id` supplies stable intended-call identity. The Broker derives
|
|
292
|
+
deterministic command and idempotency keys so a lost CLI result can be recovered
|
|
293
|
+
without redispatching the same intended call.
|
|
294
|
+
|
|
295
|
+
Cancellation is deterministic before dispatch and best effort after dispatch.
|
|
296
|
+
A connection loss or cancellation after mutation dispatch produces
|
|
297
|
+
`unknown_outcome` unless success or failure is known. The Agent must inspect
|
|
298
|
+
current browser state before retrying.
|
|
299
|
+
|
|
300
|
+
CLI `SIGINT` and `SIGTERM` request cancellation for the active command but do
|
|
301
|
+
not claim rollback.
|
|
302
|
+
|
|
303
|
+
## 14. Dialogs, Auth, Cookies, and Network
|
|
304
|
+
|
|
305
|
+
Dialogs are scoped to owning Agent state and remain pending until explicitly
|
|
306
|
+
accepted or dismissed. Browser Pilot never chooses a dialog response.
|
|
307
|
+
|
|
308
|
+
HTTP authentication credentials and network rules are scoped to the Agent
|
|
309
|
+
namespace. Credential values are not retained outside the active configuration
|
|
310
|
+
needed by Chrome.
|
|
311
|
+
|
|
312
|
+
Cookie reads are scoped to the current URL or explicit domain.
|
|
313
|
+
|
|
314
|
+
Network support includes bounded request journals, detail/body reads, blocking,
|
|
315
|
+
mocking, header overrides, rule listing/removal, and clearing. Sensitive fields
|
|
316
|
+
carry internal sensitivity metadata and bounded output.
|
|
317
|
+
|
|
318
|
+
## 15. Files and Media
|
|
319
|
+
|
|
320
|
+
CLI file-producing commands return:
|
|
321
|
+
|
|
322
|
+
- absolute local path;
|
|
323
|
+
- MIME type;
|
|
324
|
+
- byte size;
|
|
325
|
+
- width and height for images when available;
|
|
326
|
+
- command-specific metadata such as annotation count.
|
|
327
|
+
|
|
328
|
+
Output resolution order is:
|
|
329
|
+
|
|
330
|
+
1. explicit filename;
|
|
331
|
+
2. `BROWSER_PILOT_OUTPUT_DIR`;
|
|
332
|
+
3. current working directory.
|
|
333
|
+
|
|
334
|
+
The Broker stores media and downloads in private bounded temporary storage,
|
|
335
|
+
then exports them to a caller-authorized absolute path. Binary bytes are never
|
|
336
|
+
written into normal CLI JSON.
|
|
337
|
+
|
|
338
|
+
Uploads copy a caller-authorized source into private storage and release it
|
|
339
|
+
after browser assignment. Downloads copy Chrome's completed file. Browser Pilot
|
|
340
|
+
never moves, truncates, deletes, or cancels the user's original file.
|
|
341
|
+
|
|
342
|
+
Storage has per-item, per-Agent, and global quotas plus expiry. Paths are not
|
|
343
|
+
derived from external IDs, symbolic-link escapes are rejected, and file modes
|
|
344
|
+
are private.
|
|
345
|
+
|
|
346
|
+
## 16. Downloads
|
|
347
|
+
|
|
348
|
+
Download tracking is attached only to controlled sessions and browser
|
|
349
|
+
connection generation. Completed browser events are attributed to the owning
|
|
350
|
+
Agent before copying.
|
|
351
|
+
|
|
352
|
+
Requirements:
|
|
353
|
+
|
|
354
|
+
- downloads from unrelated or unowned tabs are ignored;
|
|
355
|
+
- duplicate browser GUID ownership fails closed;
|
|
356
|
+
- quota failure affects only the Browser Pilot copy;
|
|
357
|
+
- releasing Agent state stops tracking but does not cancel Chrome's download;
|
|
358
|
+
- reconnect configures browser-level download events once per generation;
|
|
359
|
+
- completed downloads can be listed and exported later by the same Agent key.
|
|
360
|
+
|
|
361
|
+
## 17. Error Contract
|
|
362
|
+
|
|
363
|
+
Non-TTY CLI failures emit one JSON object with:
|
|
364
|
+
|
|
365
|
+
- `ok: false`;
|
|
366
|
+
- stable `code`;
|
|
367
|
+
- human-readable `error`;
|
|
368
|
+
- `retryable`;
|
|
369
|
+
- optional bounded `context`;
|
|
370
|
+
- optional `remediation`;
|
|
371
|
+
- optional recovery `hint`.
|
|
372
|
+
|
|
373
|
+
Consumers branch on `code`, never English text. Stable codes cover input,
|
|
374
|
+
authorization, browser lifecycle, profile routing, target contention, stale
|
|
375
|
+
state, verification, waiting, cancellation, uncertainty, quota, and internal
|
|
376
|
+
failure.
|
|
377
|
+
|
|
378
|
+
`retryable: true` means a later call may be valid. It never means the same
|
|
379
|
+
mutation is safe to replay.
|
|
380
|
+
|
|
381
|
+
## 18. Process and Shutdown Model
|
|
382
|
+
|
|
383
|
+
The public CLI is short-lived. The per-user Broker and managed-target
|
|
384
|
+
connection supervisor are private background roles.
|
|
385
|
+
|
|
386
|
+
Startup uses a private lock and locator. Simultaneous CLIs create exactly one
|
|
387
|
+
Broker. A live but unresponsive Broker is never silently replaced.
|
|
388
|
+
|
|
389
|
+
The supervisor is the sole owner of Browser Pilot-created target identities.
|
|
390
|
+
After Broker crash or abrupt exit, it closes only managed targets and eligible
|
|
391
|
+
managed popup descendants. It never persists target IDs to disk and never
|
|
392
|
+
closes user-opened tabs.
|
|
393
|
+
|
|
394
|
+
`bp disconnect` first releases the current Agent namespace. It may stop the
|
|
395
|
+
whole Broker only when:
|
|
396
|
+
|
|
397
|
+
1. no active Agent state remains;
|
|
398
|
+
2. the Broker process identity has not changed;
|
|
399
|
+
3. the invoking executable version and content/path identity match the
|
|
400
|
+
installation that started the Broker.
|
|
401
|
+
|
|
402
|
+
A compatible CLI from another installation succeeds in releasing its state but
|
|
403
|
+
leaves the Broker running.
|
|
404
|
+
|
|
405
|
+
## 19. Distribution
|
|
406
|
+
|
|
407
|
+
Supported distributions are:
|
|
408
|
+
|
|
409
|
+
- global npm install;
|
|
410
|
+
- local npm install or `npx --no-install`;
|
|
411
|
+
- product-bundled npm package with pinned Node runtime;
|
|
412
|
+
- self-contained native executable.
|
|
413
|
+
|
|
414
|
+
Native targets are Apple Silicon macOS, x64 Linux, and x64 Windows. Intel Mac
|
|
415
|
+
is intentionally unsupported.
|
|
1076
416
|
|
|
1077
|
-
|
|
1078
|
-
|
|
417
|
+
Every release includes:
|
|
418
|
+
|
|
419
|
+
- versioned native archives;
|
|
420
|
+
- archive checksum sidecars;
|
|
421
|
+
- per-file checksums and licenses inside native archives;
|
|
422
|
+
- an Agent skill/plugin archive and checksum;
|
|
423
|
+
- a release index binding CLI version, protocol range, skill compatibility
|
|
424
|
+
range, tested version, supported platforms, asset sizes, and hashes.
|
|
425
|
+
|
|
426
|
+
CLI, skill, plugin manifests, compatibility metadata, and checksums must be
|
|
427
|
+
released from one synchronized root version.
|
|
428
|
+
|
|
429
|
+
## 20. Security and Privacy Invariants
|
|
430
|
+
|
|
431
|
+
- No browser extension is used or required.
|
|
432
|
+
- Broker runtime directories and files are private to the OS user.
|
|
433
|
+
- Public results never expose raw CDP target/session/node IDs.
|
|
434
|
+
- Cross-Agent object access fails closed.
|
|
435
|
+
- User tabs are not closed by release, crash cleanup, or bulk close.
|
|
436
|
+
- Browser authorization is never requested by discovery or timers.
|
|
437
|
+
- Output is bounded and carries truncation metadata.
|
|
438
|
+
- Password and credential values are not echoed in verification evidence.
|
|
439
|
+
- File operations reject unsafe paths and symbolic-link escapes.
|
|
440
|
+
- Raw CDP is not a public Agent interface.
|
|
441
|
+
|
|
442
|
+
## 21. Acceptance Gates
|
|
443
|
+
|
|
444
|
+
A release is acceptable only when all of the following hold:
|
|
445
|
+
|
|
446
|
+
- TypeScript build succeeds.
|
|
447
|
+
- All unit and Broker process tests pass.
|
|
448
|
+
- Isolated Chrome capability metrics do not regress.
|
|
449
|
+
- Local Playwright core, compatibility, and network suites pass.
|
|
450
|
+
- npm global, local, and product-bundled distributions pass black-box tests.
|
|
451
|
+
- Native self-contained verification passes on release platforms.
|
|
452
|
+
- CLI help exposes `--client-key`, `--request-id`, `--timeout`, status, wait,
|
|
453
|
+
command recovery, and file commands.
|
|
454
|
+
- No removed persistent adapter command is present.
|
|
455
|
+
- Two Agent keys remain isolated while sharing one Broker.
|
|
456
|
+
- Compatible CLI installations reuse a Broker.
|
|
457
|
+
- A non-owner CLI cannot stop another installation's Broker.
|
|
458
|
+
- One physical user tab is never controlled concurrently by two Agents.
|
|
459
|
+
- Browser reconnect invalidates old refs and target identities.
|
|
460
|
+
- Managed crash cleanup leaves user tabs untouched.
|
|
461
|
+
- Screenshot, PDF, download, upload, and saved network results use local files
|
|
462
|
+
with complete metadata.
|
|
463
|
+
|
|
464
|
+
Real-site canaries report drift separately and do not make deterministic release
|
|
465
|
+
gates depend on third-party uptime.
|
|
466
|
+
|
|
467
|
+
## 22. Non-Goals
|
|
468
|
+
|
|
469
|
+
- General cross-browser automation outside Chromium CDP.
|
|
470
|
+
- A remote multi-tenant browser service.
|
|
471
|
+
- Browser-extension capabilities such as tab-group management.
|
|
472
|
+
- A model provider or Agent orchestration framework.
|
|
473
|
+
- Per-action user-intent approval inside Browser Pilot.
|
|
474
|
+
- A public SDK, MCP server, or alternate machine protocol.
|