browser-pilot-cli 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,1047 +1,474 @@
1
1
  # Browser Pilot Platform Specification
2
2
 
3
- Status: **Authoritative, approved for implementation**
4
- Baseline: `0c5661ccb133b2c9feda15732892f66ce947f232`
5
- Last updated: 2026-07-25
6
-
7
- This document is the source of truth for turning Browser Pilot from a
8
- single-agent CLI with global state into a user-level browser control runtime
9
- that any local Agent can install or embed. The two execution plans in
10
- `docs/plans/` derive from this specification.
11
-
12
- ## Purpose and Reader
13
-
14
- This specification is for Browser Pilot maintainers, Agent-product integrators,
15
- security reviewers, and test authors. It defines observable behavior and
16
- compatibility boundaries before implementation. Internal component names near
17
- the end are implementation boundaries required to preserve those behaviors;
18
- they are not third-party APIs.
19
-
20
- ## Evidence and Assumptions
21
-
22
- Evidence from the baseline repository:
23
-
24
- - The npm package currently publishes CLI binaries but no library exports.
25
- - High-level actions, output formatting, file writes, and process exits are
26
- combined in `src/cli.ts`.
27
- - `src/state.ts` persists one active target, CDP session, and frame context.
28
- - `src/snapshot.ts` persists one target's refs in a global file.
29
- - `src/daemon.ts` exposes arbitrary CDP and keeps auth, network, dialog, and
30
- popup state globally.
31
- - Existing fixtures prove the current browser operations are worth preserving.
32
-
33
- Evidence from the browser-use comparison is recorded in
34
- `docs/plans/browser-capability-evolution.md` at the pinned research commit.
35
-
36
- Assumptions used by this specification:
37
-
38
- - Supported Agent runtimes can spawn a local child process and read protected
39
- local Artifact files.
40
- - Browser Pilot and embedded Agents execute under the same OS user account.
41
- - Chrome continues to require an explicit user action to enable or authorize
42
- remote debugging.
43
- - An Agent product can map its own active lifecycle to Workspace and Lease
44
- without Broker knowledge of that product's domain nouns.
45
-
46
- ## Decision Summary
47
-
48
- - **DEC-1:** Browser Pilot is an Agent-neutral, per-OS-user Browser Broker. It
49
- is not a Tenon component and does not model conversations, runs, tasks, or
50
- workers.
51
- - **DEC-2:** Root Chrome DevTools Protocol (CDP) is the only browser connector.
52
- Browser extensions are never required or supported.
53
- - **DEC-3:** The public product surface is one official CLI executable. It
54
- supports human one-shot commands and a persistent `bridge --stdio` machine
55
- mode. Native SDK and MCP surfaces are out of scope.
56
- - **DEC-4:** Existing one-shot `bp` commands remain a first-class integration
57
- path. Embedded products bundle the same executable and use the stdio bridge.
58
- - **DEC-5:** The production machine interface exposes controlled browser tools,
59
- not arbitrary CDP forwarding. Raw CDP is internal. JavaScript evaluation is
60
- an explicit developer capability.
61
- - **DEC-6:** Invoking Browser Pilot grants the Agent full control of all
62
- eligible current and future tabs in the selected BrowserInstance. Browser
63
- Pilot does not implement task-intent approval, selected-tab grants, or
64
- per-action confirmation. An Agent host may remove operations when launching
65
- the Broker, and owns any higher-level approval UX it wants.
66
- - **DEC-7:** Browser profile state is shared because controlled targets run in
67
- the user's browser profile. Target assignments, commands, refs, rules, and
68
- artifacts are isolated by Broker objects.
69
-
70
- ## Objective, Constraints, and Selected Target
71
-
72
- - **OBJ-1:** Any local Agent that can execute a process must be able to control
73
- the user's real browser reliably, while preserving the user's profiles,
74
- logins, cookies, and installed extensions.
75
- - **OBJ-2:** Multiple unrelated Agent products must be able to use Browser Pilot
76
- concurrently without overwriting each other's target, frame, ref, auth, or
77
- network state.
78
- - **OBJ-3:** Agents that already install and call `bp` themselves must continue
79
- to work after the platform migration.
80
- - **Minimum acceptable outcome:** The compatibility requirements in this spec
81
- all pass. A partial SDK, Tenon-only adapter, or globally shared target state
82
- does not count.
83
- - **Clean-slate answer:** A signed, self-contained executable with a local
84
- Broker, versioned stdio protocol, complete browser control, strong runtime
85
- isolation, and a conformance kit.
86
- - **Selected brownfield target:** Evolve the current TypeScript/npm CLI into
87
- that architecture without replacing proven CDP capabilities or breaking the
88
- current command workflow.
89
-
90
- ### Constraints
91
-
92
- - **CON-1 hard:** No browser extension may be used.
93
- - **CON-2 hard:** All eligible ordinary Chrome tabs in the selected
94
- BrowserInstance are visible and controllable. DevTools, unsupported internal
95
- pages, and Browser Pilot's own internal targets remain excluded.
96
- - **CON-3 hard:** Browser Pilot must not depend on Agent-specific lifecycle
97
- concepts or model-provider APIs.
98
- - **CON-4 hard:** Production clients cannot invoke arbitrary root CDP methods.
99
- - **CON-5 hard:** Sensitive browser data and media must not be emitted to logs
100
- or world-readable files.
101
- - **CON-6 legacy:** Existing `bp` commands, JSON output, npm installation, and
102
- the current Agent skill must remain usable during migration.
103
- - **CON-7 platform:** Without an extension, root CDP exposes the browser to any
104
- process that can independently reach its debugging endpoint. Broker
105
- isolation protects Broker clients from each other; it is not an OS sandbox.
106
- - **CON-8 scope:** The universal integration target is local Agents that can
107
- spawn a process and access local files. Hosted Agents without a local
108
- execution bridge are not supported by CLI-only distribution.
109
-
110
- ### Rejected Alternatives
111
-
112
- - **OPT-1 MCP as the canonical interface:** Rejected because it creates a
113
- second public product surface and runtime dependency. An Agent adapter may
114
- translate the stdio bridge to MCP outside Browser Pilot.
115
- - **OPT-2 Native SDKs:** Rejected because every language binding would create a
116
- versioned compatibility obligation. The executable protocol is the SDK.
117
- - **OPT-3 One-shot CLI only:** Rejected because it cannot efficiently deliver
118
- events, cancellation, leases, or repeated tool calls to an embedded Agent.
119
- - **OPT-4 Agent-specific embedding:** Rejected because Tenon and OpenClaw are
120
- clients and conformance fixtures, not Broker domain concepts.
121
-
122
- ### Revisit Triggers
123
-
124
- Reconsider MCP or a remote protocol only if Browser Pilot must support Agents
125
- that cannot spawn a local process. Reconsider SDKs only if the stdio protocol
126
- cannot provide required performance or type safety after measured use.
127
-
128
- ## Preserved and Changed Behavior
129
-
130
- ### Preserved
131
-
132
- - Agent-managed installation via npm and direct `bp` invocation.
133
- - Chrome remote-debugging discovery and connection.
134
- - A distinct visible Pilot window using the user's browser profile.
135
- - AX snapshots and numbered refs.
136
- - Navigation, click, type, keyboard, upload, tabs, frames, dialogs, HTTP auth,
137
- screenshots, PDF, cookies, and network tooling.
138
- - Human-readable terminal output and JSON output for scripts.
139
- - The existing skill and Playwright fixture coverage.
140
-
141
- ### Changed
142
-
143
- - CLI handlers become presentation adapters over reusable Broker services.
144
- - Global active target/session/frame state becomes isolated Workspace and Lease
145
- state.
146
- - The single refs file becomes observation-scoped Broker state.
147
- - The daemon's arbitrary `/cdp` endpoint becomes private implementation wiring.
148
- - Dialogs become explicit events and decisions; they are not auto-accepted.
149
- - Popup adoption requires a verified owned opener chain.
150
- - Existing user tabs and newly opened tabs are included automatically in the
151
- Workspace inventory; invoking the tool is the authorization boundary.
152
- - Auth, network rules, request journals, and artifacts are scoped and cleaned up
153
- by the owning Workspace or Lease.
154
- - Screenshots and PDFs become mode-restricted, owner-scoped Artifact results
155
- with quotas and expiry. The human CLI may export an Artifact to a
156
- caller-selected path.
157
-
158
- ## Product Model
159
-
160
- ### ClientPrincipal
161
-
162
- A stable identity for an installed Agent product or direct CLI user. It owns
163
- Workspaces and their transient resources. It is used for correlation and
164
- isolation, not as proof that an Agent's semantic decision is correct. A
165
- Principal is not a conversation or task.
166
-
167
- ### ClientConnection
168
-
169
- One live `bridge --stdio` process connection. It is authenticated by the local
170
- Broker and associated with exactly one ClientPrincipal after initialization.
171
-
172
- ### BrowserInstance
173
-
174
- One running, discoverable Chromium-family browser endpoint identified by a
175
- process-stable identity, product, user-data root, debugging endpoint, and
176
- connection generation. One endpoint may expose several simultaneously live
177
- Chrome Profile contexts; those are not separate BrowserInstances.
178
-
179
- ### ProfileContext
180
-
181
- One live regular browser context inside a BrowserInstance. Its public
182
- `profileContextId` is opaque and scoped to the current browser connection
183
- generation; the raw CDP browser-context ID is internal. A ProfileContext has a
184
- connection-scoped neutral label, optional verified display metadata, bounded
185
- representative tabs, and current tab counts. Passive discovery never opens a
186
- tab to infer its display name. Reconnect invalidates every ProfileContext ID.
187
-
188
- ### BrowserWorkspace
189
-
190
- A logical browser working set owned by one ClientPrincipal. It owns one or more
191
- ManagedTabSets, an optional selected ProfileContext, and transient scoped state.
192
- Workspaces live in Broker memory by default and are recreated after Broker
193
- restart; Agent adapters map their own active lifecycle to this object. Profile
194
- selection is never persisted as a global browser default.
195
-
196
- ### ManagedTabSet
197
-
198
- The logical collection of Broker-created tabs used for isolated Agent work. It
199
- is normally backed by a dedicated visible Pilot window because Browser Pilot
200
- does not use extension-only native tab-group APIs. New task navigation and
201
- owned popups stay in this set by default. A ManagedTabSet is permanently bound
202
- to at most one ProfileContext; a Workspace that creates managed work in multiple
203
- Profiles owns one set per Profile. Workspace cleanup may close these sets, but
204
- never user-owned tabs outside them.
3
+ Status: active architecture for the next release line.
205
4
 
206
- ### ControlLease
5
+ ## 1. Product Definition
207
6
 
208
- A time-bounded command session in a Workspace. It carries negotiated
209
- capabilities, heartbeat expiry, and explicit target control assignments. Agent
210
- adapters map their own active work lifecycle to this object.
7
+ Browser Pilot gives a local AI Agent controlled access to the user's eligible
8
+ Chromium tabs through a command-line interface. It reuses the user's live
9
+ profiles, logins, cookies, and extensions without a browser extension.
211
10
 
212
- ### BrowserControlPolicy
11
+ The product is not an Agent runtime. It does not decide user intent, add
12
+ per-action approvals, own the model conversation, or expose a second native
13
+ tool protocol.
213
14
 
214
- An optional launch-time reduction of Browser Pilot operations configured by
215
- the Agent host, for example disabling `network.modify` or `tabs.close`. The
216
- default policy allows every supported operation. This is tool exposure, not a
217
- Browser Pilot approval lifecycle, and it cannot be expanded through a tool call
218
- inside the running bridge.
15
+ ## 2. Architectural Decision
219
16
 
220
- ### ControlledTarget
17
+ There is one public Agent integration path:
221
18
 
222
- A target addressable by a Workspace. Its origin is `managed`, `managed_popup`,
223
- or `user_tab`, and it records its ProfileContext and an optional ManagedTabSet.
224
- A user tab retains user
225
- ownership: releasing a Workspace removes Broker mappings without closing the
226
- tab. Every result identifies its Workspace, Lease, opaque controlled-target ID,
227
- URL, origin, and browser connection generation. Raw CDP target IDs are not a
228
- public identifier.
19
+ ```text
20
+ Agent
21
+ -> Browser Pilot skill
22
+ -> host's existing shell/command runner
23
+ -> short-lived bp CLI process
24
+ -> shared per-user Browser Pilot Broker
25
+ -> supervised CDP connection
26
+ -> Chrome
27
+ ```
229
28
 
230
- ### Observation
29
+ Browser Pilot does not publish or support:
231
30
 
232
- An immutable, bounded view of a ControlledTarget at a point in time. Public element
233
- handles are `observationId + ref`. Observations are never shared between
234
- Workspaces.
31
+ - a browser extension;
32
+ - an MCP server;
33
+ - a language-specific Native SDK;
34
+ - Browser Pilot-specific native Agent tools;
35
+ - a public JSON-RPC or stdio adapter;
36
+ - imports from `src/*`;
37
+ - direct consumer access to Broker sockets or locator files.
235
38
 
236
- ### AgentHint
39
+ This decision applies equally to Agent-managed installs and products that
40
+ bundle Browser Pilot. A product makes the CLI available on the Agent command
41
+ environment's `PATH`, installs the same skill, and injects stable environment.
237
42
 
238
- A bounded, discriminated browser-state signal attached to an Observation or
239
- event. A hint has a stable `code`, `source`, `confidence`, and
240
- `recommendedAction`, plus code-specific public data. Hints report browser
241
- evidence and a useful next inspection strategy; they never approve an action,
242
- assert that the task succeeded, or replace fresh Observation and event state.
43
+ ## 3. Public Contract
243
44
 
244
- ### Command
45
+ The supported public contract consists of:
245
46
 
246
- A requested operation with a unique command ID, idempotency key, deadline,
247
- capability requirement, and target assignment. Each target serializes its
248
- commands through one actor.
47
+ 1. The `bp` and `browser-pilot` executable names.
48
+ 2. Command syntax and `--help` output.
49
+ 3. JSON success and failure results.
50
+ 4. Stable machine error codes.
51
+ 5. Local file results and their metadata.
52
+ 6. Environment variables documented in this repository.
53
+ 7. Skill files and their declared CLI compatibility range.
54
+ 8. Versioned native archives, npm package, checksums, and release index.
249
55
 
250
- ### Artifact
56
+ Internal Broker methods, schemas, IDs, sockets, files, CDP sessions, and
57
+ process roles are not public APIs. They may evolve without creating a second
58
+ integration surface as long as the CLI contract remains compatible.
251
59
 
252
- A Broker-managed binary result such as a screenshot, preview, PDF, or download.
253
- Artifacts have metadata, sensitivity, ownership, size limits, retention state,
254
- and expiry.
60
+ ## 4. Host Integration Contract
255
61
 
256
- ### BrowserEvent
62
+ A shell-capable Agent host needs only:
257
63
 
258
- A typed, ordered event in a bounded per-Workspace journal. Events can be pushed
259
- over stdio and replayed with a cursor. Every event carries the browser connection
260
- generation of its source. Ordinary events from a retired generation are dropped;
261
- Command terminal state and explicit reconnect cleanup remain replayable with
262
- their historical generation.
64
+ - a normal local command execution tool;
65
+ - the complete Browser Pilot skill;
66
+ - a compatible `bp` executable on the command environment's `PATH`;
67
+ - a stable `BROWSER_PILOT_CLIENT_KEY` per independent Agent;
68
+ - an absolute task-owned `BROWSER_PILOT_OUTPUT_DIR` when file results are used;
69
+ - ordinary access to returned task files.
263
70
 
264
- ## Public Executable Surface
71
+ The host must not map every Browser Pilot command into a native tool. The skill
72
+ provides progressive disclosure, and CLI help provides exact runtime command
73
+ discovery.
265
74
 
266
- The official distribution exposes one executable under both `browser-pilot`
267
- and the compatible `bp` alias.
75
+ The host should pin the CLI version tested by its product release. The skill
76
+ declares a compatible range so a compatible user-installed command can also be
77
+ used. The host must update its executable, skill, compatibility metadata, and
78
+ checksums together.
268
79
 
269
- ```text
270
- browser-pilot
271
- broker serve
272
- bridge --stdio
273
- tool list
274
- tool call
275
- artifact export|release
276
- browsers|connect|disconnect|open|snapshot|click|type|keyboard|press
277
- read|eval|upload|screenshot|pdf|cookies|frame|auth|tabs|tab|close|net
278
- ```
80
+ ## 5. Browser Scope and Authorization
279
81
 
280
- Three distribution modes are equivalent:
281
-
282
- 1. User or Agent global installation.
283
- 2. Project-local installation and `npx browser-pilot` invocation.
284
- 3. Product-bundled official executable launched by an Agent adapter.
285
-
286
- Embedded products must not import `src/*`, depend on daemon socket details, or
287
- parse human-readable output.
288
-
289
- The npm distribution contains bundled JavaScript entry points for the CLI,
290
- Broker, and managed-target janitor and requires Node 18 or newer. The native
291
- distribution is one Node SEA executable containing the runtime and JavaScript
292
- dependencies. It re-executes itself with reserved private role arguments for
293
- Broker and janitor processes; it neither downloads a runtime nor falls back to
294
- system Node. Both layouts derive executable installation identity from the
295
- public entry point, so private child roles do not create false version
296
- boundaries.
297
-
298
- Native artifacts are built and verified on their target OS and architecture.
299
- Each archive includes an actual-state signature descriptor, file sizes and
300
- SHA-256 hashes, runtime/dependency licenses, and an archive checksum. macOS uses
301
- Developer ID signing with hardened runtime and explicit JIT entitlements when
302
- credentials exist, otherwise ad-hoc signing. Windows uses Authenticode when a
303
- certificate exists, otherwise records unsigned state. Linux records unsigned
304
- state. Build or release automation must fail on incomplete credentials and
305
- must never describe an artifact as signed when signing did not run.
306
-
307
- ### Broker Locator and Startup
308
-
309
- The executable, not an embedding Host, resolves the per-user Broker endpoint.
310
- On Unix it uses an owner-only domain socket and falls back to a deterministic
311
- short owner-specific runtime directory when the state path would exceed
312
- portable socket limits. On Windows it uses a named pipe derived from the OS
313
- user identity. Persistent state defaults to the Browser Pilot user directory
314
- (`%LOCALAPPDATA%\\Browser Pilot` on Windows). An explicit absolute
315
- `BROWSER_PILOT_HOME` is the cross-platform installation-level isolation
316
- override. It is never selected automatically.
317
-
318
- State/runtime directories are mode `0700` and regular metadata files are mode
319
- `0600` where POSIX modes apply. The socket is mode `0600`; Windows relies on the
320
- creating user's process-token DACL for its local named pipe. Symlinked final
321
- directories and metadata, foreign ownership, unexpected file types, and
322
- overly-permissive metadata are rejected.
323
-
324
- The versioned locator records PID, endpoint/transport, start time, Broker
325
- process identity, service version, executable installation identity/path, and
326
- the supported protocol range. A separate owner-only `broker-versions.json`
327
- keeps only the current and immediately previous executable metadata for
328
- diagnosis and explicit rollback; it does not preserve an executable or any
329
- browser, target, Workspace, Lease, ref, credential, rule, or command state. An
330
- atomic owner-recorded startup lock surrounds discovery and daemon launch. Every
331
- contender health-checks before and after the lock; therefore exactly one
332
- process starts the Broker and all others reuse it. Broker startup and discovery
333
- are passive and do not open a browser WebSocket.
334
- A lock whose owner is dead can be reclaimed; a live owner is never displaced
335
- merely because startup is slow. During Broker startup, the daemon writes an
336
- owner-only `starting` process record. Chrome authorization begins only after an
337
- explicit `browser.connect`; all concurrent callers share its single in-flight
338
- connection attempt. A dead
339
- locator/socket is removed without signaling its recorded PID. If a ready PID is
340
- alive but its endpoint is unresponsive, Browser Pilot returns structured
341
- restart remediation and never silently kills or replaces the process.
342
-
343
- ## Primary Flows
344
-
345
- ### FLOW-1 Agent-managed one-shot use
346
-
347
- An Agent installs Browser Pilot globally or in its project and runs `bp connect`.
348
- The CLI starts or reuses the compatible per-user Broker, creates its
349
- compatibility Workspace and Lease, then explicitly requests the browser
350
- connection. Other browser commands require that connection and never trigger
351
- authorization implicitly.
352
- Existing and future eligible user tabs are available immediately. The command
353
- returns the existing JSON shape during migration. Failures include a stable
354
- machine code in addition to compatible human guidance. One-shot processes use
355
- a fixed daemon-internal Connection, idempotently keyed Workspace, and renewable
356
- Lease with a maximum five-minute TTL. Normal process exit leaves that transient
357
- daemon-memory state available to the next command; expiry invalidates refs and
358
- releases target control. No target, frame, session, Observation, ref, auth, or
359
- network mapping is persisted to disk. `bp disconnect` explicitly releases the
360
- compatibility Workspace and closes only its managed targets. It stops the
361
- daemon only when called by the matching executable installation and no live
362
- embedded Connection remains; otherwise it returns `protocol_incompatible` or
363
- `broker_in_use` without replacing or terminating the Broker. JavaScript dialogs
364
- remain pending and are handled explicitly with `bp dialogs` and `bp dialog`;
365
- Workspace isolation prevents access to other clients' dialogs.
366
-
367
- ### FLOW-2 Product-embedded use
368
-
369
- An Agent product bundles or resolves the official executable and starts
370
- `browser-pilot bridge --stdio`. It initializes, negotiates protocol and
371
- capabilities, creates or resumes a Workspace, acquires a Lease, discovers tools,
372
- calls `browser.connect` when the selected browser is not ready, and maps the
373
- remaining tools into its own runtime. Bridge launch, initialization, discovery,
374
- and Workspace creation stay passive. It calls tools, consumes ordered
375
- events, converts Artifact files into native media content, heartbeats while
376
- active, then releases the Lease. A process crash is recovered by Lease expiry.
377
-
378
- ### FLOW-3 Browser disconnect and recovery
379
-
380
- The Broker emits connection loss, stops dispatching target commands, and marks
381
- commands whose effects are uncertain as `unknown_outcome`. It never reconnects
382
- on a timer. A client explicitly calls `browser.connect` when recovery is
383
- appropriate. After that succeeds, the Broker
384
- creates a new browser connection generation, invalidates old CDP sessions,
385
- ControlledTarget mappings, frames, and Observations, and emits structured recovery
386
- state. The daemon rediscovers only the originally selected browser profile; it
387
- does not silently switch to another running browser. Consumers rebuild inventory
388
- with `browser.tabs.list`. Browser Pilot never replays a mutating command
389
- automatically.
390
-
391
- ### FLOW-4 Concurrent clients
392
-
393
- Two unrelated Principals create separate Workspaces and Pilot targets. Their
394
- actions may execute concurrently on different target actors. Any attempt to use
395
- the other Principal's target, ref, rule, Artifact, or Workspace fails closed.
396
- Releasing or crashing one client does not close or mutate the other's targets.
397
-
398
- ### FLOW-5 Existing and future user tabs
399
-
400
- When a Host exposes Browser Pilot to an Agent, that invocation authorizes the
401
- Agent to use the selected BrowserInstance. `tabs.list` immediately returns the
402
- Workspace's ManagedTabSets plus all eligible user tabs, tagged by origin. Tabs
403
- opened later appear on the next inventory refresh. The Agent may switch to and
404
- operate any listed tab, including a form the user opened before invoking it.
405
-
406
- Browser Pilot does not ask the Agent to request access and does not display a
407
- grant UI. If a product needs confirmation or restricted tool exposure, the
408
- Agent host implements that policy before launching the bridge or maps it to a
409
- launch-time BrowserControlPolicy. Releasing a Lease or Workspace removes
410
- transient mappings and rules but leaves user tabs open. Closing a user tab
411
- always requires an explicit target-specific command; bulk cleanup applies only
412
- to ManagedTabSets.
413
-
414
- ### FLOW-6 Multiple live Chrome Profiles
415
-
416
- `browser.tabs.list` inventories eligible tabs from every Profile context exposed
417
- by the selected BrowserInstance. Existing targets are immediately controllable
418
- and carry an opaque `profileContextId`. New managed work resolves a Profile from
419
- an explicit context, the current active target, the Workspace selection, or a
420
- single available context, in that order. Multiple contexts without an anchor
421
- return `profile_selection_required` before creating a target. The Agent host
422
- asks the user which Profile to use; Browser Pilot does not add an approval UI.
423
-
424
- `browser.profiles.list` is passive and returns bounded, connection-scoped
425
- summaries. `browser.profiles.select` updates only the owning Workspace. A Chrome
426
- reconnect invalidates old ProfileContext IDs and requires relisting and
427
- reselection. Selection clears only the Lease's logical active-target anchor; it
428
- does not release control, activate a tab, or change Chrome focus. The exact
429
- creation and compatibility algorithm is frozen in `docs/plans/profile-context-routing.md`.
430
-
431
- ## Machine Protocol
432
-
433
- `bridge --stdio` uses JSON-RPC 2.0 over newline-delimited UTF-8 JSON. Each line
434
- contains one complete message. Stdout is protocol-only. Logs go to stderr and
435
- must redact secrets. The bridge terminates on malformed framing after returning
436
- a protocol error when possible.
437
-
438
- Required methods:
82
+ Browser Pilot exposes all eligible page tabs on the authorized browser
83
+ endpoint:
439
84
 
440
- ```text
441
- initialize
442
- tools/list
443
- tools/call
444
- workspaces/create|get|release
445
- leases/create|heartbeat|release
446
- commands/cancel|get
447
- events/poll
448
- artifacts/import|get|export|retain|release
449
- shutdown
450
- ```
85
+ - managed tabs created by Browser Pilot;
86
+ - eligible popup descendants of managed tabs;
87
+ - eligible tabs opened by the user in every live profile.
451
88
 
452
- Asynchronous events use best-effort `events/event` JSON-RPC notifications.
453
- Polling with a cursor is the authoritative recovery path after reconnect,
454
- notification loss, or backpressure. Notifications and responses may interleave.
455
-
456
- ### Initialization
457
-
458
- The client sends its product identity, instance identity, supported protocol
459
- range, requested capabilities, launch mode, and optional protocol 1.1 transport
460
- limits. The response includes:
461
-
462
- - service and executable versions;
463
- - selected protocol version;
464
- - supported and granted capabilities;
465
- - Broker process identity and connection ID;
466
- - browser discovery and authorization state;
467
- - transport and result size limits.
468
-
469
- Initialization must fail with a structured incompatibility error when no
470
- protocol overlap exists. A newer executable must not silently replace an
471
- incompatible Broker that has live clients.
472
-
473
- Protocol 1.0 clients use the service transport ceilings. Protocol 1.1 clients
474
- may declare maximum message and result byte sizes; the Broker selects the lower
475
- client/service value per Connection. Initialization itself uses fixed bootstrap
476
- ceilings, and pipelined messages after initialize are not parsed until the
477
- selected limits take effect. Artifact and journal capacities remain service
478
- resource limits. Pending ordinary and out-of-band control calls have separate
479
- bounds so overload cannot create an unbounded Promise queue or prevent dialog
480
- and cancellation control from overtaking a blocked action.
481
-
482
- Protocol 1.2 adds connection-generation-scoped ProfileContext discovery and
483
- selection, Profile identity on tab inventory, and context-aware managed target
484
- creation. Protocol 1.0/1.1 clients retain existing-target control and automatic
485
- single-context behavior, but an ambiguous multi-context open fails rather than
486
- silently selecting a Profile.
487
-
488
- ### Tool Contract
489
-
490
- `tools/list` is the canonical machine-readable manifest. Every tool declares:
491
-
492
- - stable name and description;
493
- - JSON input schema;
494
- - output schema and possible Artifact kinds;
495
- - required capabilities;
496
- - whether it may mutate browser state;
497
- - cancellation and idempotency semantics;
498
- - sensitivity classification.
499
-
500
- The tool-level `sensitivity.input` and `sensitivity.output` arrays declare every
501
- classification that may occur in that direction. Content-bearing input and
502
- output schema nodes use `x-browser-pilot-sensitivity` to identify the specific
503
- fields an adapter must taint when it maps values into model text, image, or file
504
- content. The manifest validator rejects unknown, duplicate, or field-level
505
- classifications missing from the tool-level declaration. Schema annotations do
506
- not change the call or result value shape.
507
-
508
- `tools/call` returns `{ command, result?, error? }`; each manifest output schema
509
- describes the inner `result`. Callers may provide an opaque Command ID,
510
- idempotency key, and relative deadline. `commands/get` and `commands/cancel`
511
- accept a Command ID plus its Workspace ID when applicable. They authorize by
512
- ClientPrincipal ownership so a replacement Connection can inspect known state
513
- without inheriting the original Lease.
514
-
515
- Action results include bounded typed evidence where Browser Pilot can observe
516
- an effect. Ref clicks combine local control state and composed focus changes
517
- with Broker-owned navigation, normalized document, dialog, and attributable
518
- popup signals. `verified` means at least one supported effect was observed,
519
- `mismatch` means an observable control did not reach its expected state, and
520
- `unavailable` means dispatch completed without conclusive evidence. Evidence is
521
- not a claim that the Agent's higher-level business goal succeeded.
522
- Input evidence is discriminated as `type` or `keyboard`; press evidence uses the
523
- focused backend node plus bounded control-state signatures and Broker-owned
524
- signals. Upload evidence reads the selected file count and compares its browser
525
- filename without returning the local source path.
526
-
527
- Observation-producing tools also return `hints`. Observation-derived codes
528
- cover explicit autocomplete surfaces, modal overlays, explicit filter controls,
529
- and authentication surface transitions; action results may add a repeated
530
- observable no-progress hint. Hint refs are numbered refs from that Observation
531
- only. Page collection returns counts and refs, never page text, input values,
532
- passwords, selectors, raw nodes, or CDP identifiers for this classification.
533
- Blocked main-document responses and download lifecycle hints are event-derived.
534
-
535
- `hints` is an additive schema field under protocol 1.0 and 1.1. The built-in
536
- Browser tool implementation always returns the array, including an empty array;
537
- the manifest leaves the field optional so compatible older executors remain
538
- valid. Existing clients already ignore unknown response fields. No new
539
- capability is negotiated because hints expose no operation and contain only
540
- bounded derivatives of browser data already authorized by `observation.read`
541
- or `event.read`.
542
-
543
- Composite input actions pin the ControlledTarget, CDP session, selected frame,
544
- loader, Document identity, and browser connection generation for the lifetime
545
- of the action. After focus is established, `type` and `keyboard` also pin the
546
- deep composed active element. Browser Pilot checks this identity before every
547
- remaining semantic step, including select-all, deletion, each keyboard
548
- character, and optional submit. A change before the first browser mutation is
549
- a retryable `action_not_verified`; a change after any step was dispatched is
550
- `unknown_outcome` with bounded `action`, `step`, `reason`, `dispatchedSteps`,
551
- and `remainingStepsStopped` context. No remaining step is sent to the changed
552
- page and the command is never replayed automatically.
553
-
554
- The initial controlled tool families are browser discovery/status, Workspace
555
- and Lease lifecycle, open/observe/read, click/type/keyboard/press,
556
- capture/upload, tabs/frames, dialogs, auth/cookies, network, events, and
557
- Artifacts. Raw CDP is never listed. `eval` requires `developer.eval`.
558
-
559
- ## Access and Isolation Rules
560
-
561
- - **BR-1:** A public command can address a ControlledTarget only through its
562
- Workspace and a valid ControlLease.
563
- - **BR-2:** Managed targets enter a Workspace through Broker creation and a
564
- verified managed-popup opener chain. Eligible user tabs enter through target
565
- inventory. Public commands use opaque ControlledTarget IDs; Agent-supplied
566
- raw CDP identifiers are never accepted as public handles.
567
- - **BR-3:** Closing or releasing a Workspace may close its ManagedTabSets, but
568
- must not close user tabs merely because they were visible or controlled. It
569
- affects only its own mappings, rules, credentials, journals, and Artifacts.
570
- - **BR-4:** Network and auth configuration is not global. Precedence and cleanup
571
- are deterministic within the owning Workspace.
572
- - **BR-5:** Each physical browser target serializes state-changing commands.
573
- Reads that depend on document state join that same actor even when separate
574
- Workspaces address the target through different opaque ControlledTarget IDs.
575
- - **BR-6:** A physical target has at most one controlling Lease at a time.
576
- Multiple Workspaces may inventory it under different opaque IDs, but a
577
- conflicting acquisition reports `target_busy` and never silently steals
578
- control. The controlling Lease may call `browser.tabs.release` with its own
579
- opaque target ID. The Broker retires its target session and scoped state
580
- before a different Lease can acquire control with that Lease's own opaque ID.
581
- No handoff call accepts a foreign Lease or target ID.
582
- - **BR-7:** The Pilot window visibly identifies the controlling client without
583
- injecting mutable content into the page DOM.
584
- - **BR-21:** Exposing the CLI or bridge to an Agent is the product-level
585
- browser-control authorization boundary. Chrome's own remote-debugging
586
- authorization is requested only by explicit `bp connect`/`browser.connect`.
587
- Browser Pilot does not infer task intent, ask for per-tab grants, or perform
588
- per-action confirmation.
589
- - **BR-22:** The default BrowserControlPolicy permits all supported operations.
590
- A Host may remove operations at launch; a running Agent cannot expand that
591
- fixed policy through the browser tool protocol.
592
- - **BR-23:** Every Workspace can inventory the same eligible user tab under a
593
- distinct opaque ID. The single controlling-Lease rule in **BR-6** resolves
594
- concurrent mutation.
595
- - **BR-24:** Releasing a Lease or Workspace removes its user-target mappings and
596
- invalidates their Observations without closing those tabs.
597
- - **BR-25:** Eligible popups created from user tabs appear as ordinary user tabs.
598
- Managed popup inheritance remains restricted to a complete verified opener
599
- chain rooted in the same ManagedTabSet.
600
- - **BR-26:** `tabs.list` returns eligible current and future tabs, tagging each
601
- with origin and ManagedTabSet membership. Browser Pilot internal targets,
602
- DevTools, and unsupported browser-internal targets remain excluded.
603
- - **BR-27:** Bulk cleanup and `close --all` default to the ManagedTabSet. No
604
- bulk operation closes user tabs outside it. Each user tab requires an
605
- explicit target-specific close command.
606
- - **BR-29:** Runtime Profile contexts are discovered from live target metadata,
607
- not only `Target.getBrowserContexts`. Public ProfileContext IDs never expose
608
- raw CDP identifiers and are invalid after browser reconnect.
609
- - **BR-30:** Profile listing is passive. It cannot create, attach, activate, or
610
- navigate a target solely to infer a Profile name. Unverified names are
611
- omitted instead of guessed.
612
- - **BR-31:** New-target Profile routing follows explicit context, active target,
613
- Workspace selection, then single available context. Multiple unanchored
614
- contexts fail with `profile_selection_required` before browser dispatch.
615
- - **BR-32:** Every ManagedTabSet is bound to at most one ProfileContext. Managed
616
- target creation verifies the returned context before public registration, and
617
- fallback-created targets enter janitor ownership before they become
618
- addressable. Fallback ownership requires the exact new URL marker plus either
619
- the expected opener ID or exact readback of a per-command random window name;
620
- Profile membership alone is never sufficient.
621
- - **BR-33:** Switching to an existing target may update the owning Workspace's
622
- selected ProfileContext, but never prevents inventory or control of targets
623
- in other Profiles. No Profile selection is a permission grant.
624
-
625
- ## Command Reliability
626
-
627
- Mutating commands follow this runtime state model:
89
+ Browser-internal pages, extension-owned targets, and targets that cannot be
90
+ controlled safely are excluded.
628
91
 
629
- ```text
630
- accepted -> dispatched -> completed
631
- \-> unknown_outcome
632
- accepted -> cancelled
633
- accepted -> expired
634
- ```
92
+ Chrome tab-group membership does not affect tab eligibility. Browser Pilot can
93
+ control an eligible tab in an expanded or collapsed group but, without an
94
+ extension, cannot inspect, create, rename, move, collapse, or delete groups.
635
95
 
636
- - **BR-8:** Every command has a caller-provided or Broker-generated idempotency
637
- key scoped to Principal and Workspace.
638
- - **BR-9:** A duplicate completed command returns its recorded result. A
639
- duplicate dispatched command returns its status and is not dispatched again.
640
- - **BR-10:** If the Broker loses certainty after dispatch, it reports
641
- `unknown_outcome`. Mutating commands are never automatically replayed.
642
- - **BR-11:** Cancellation before dispatch prevents execution. Cancellation
643
- after dispatch is best-effort and must not claim rollback.
644
- - **BR-12:** Browser disconnect and explicit reconnect change the BrowserInstance
645
- connection generation and invalidate sessions and observations. The generation
646
- advances on successful restoration, not merely on an attempt. Discovery and
647
- timers never initiate a browser connection.
648
-
649
- Commands capture that generation when accepted. The Broker verifies it again
650
- before browser dispatch and after the executor returns. A queued stale command
651
- never reaches the browser. If a mutating command was already dispatched when the
652
- generation changed, its terminal state is `unknown_outcome`; a stale read fails
653
- with retryable `browser_disconnected` rather than returning old data.
654
-
655
- The dispatch transition occurs immediately before the first browser mutation,
656
- after argument, ref, target, and continuity preflight. Composite-action guards
657
- preserve their structured interruption context when the terminal state is
658
- `unknown_outcome`; callers must obtain fresh target/frame/Observation state
659
- before deciding what remains to be done.
660
-
661
- Normal bridge calls preserve dispatch order. `commands/get`, `commands/cancel`,
662
- and dialog list/respond calls may bypass a pending `tools/call`; JSON-RPC clients
663
- therefore correlate responses by ID rather than relying on response order. The
664
- dialog exception prevents a paused browser command from blocking the explicit
665
- response needed to release its JavaScript dialog.
666
-
667
- Workspaces, Leases, target mappings, refs, command status, idempotency records,
668
- and bounded event journals live in Broker memory by default. Broker restart
669
- invalidates them; clients initialize again, list tabs again, and observe again.
670
- Only installation identity, browser preference, Broker locator/version data,
671
- and explicit user configuration may persist. DOM snapshots, refs, cookies,
672
- passwords, network bodies, and transient control state never persist.
673
-
674
- Broker-created targets cross one additional private ownership boundary. A
675
- janitor child process is the sole owner of the root CDP WebSocket and proxies
676
- all daemon CDP requests, results, and events over private IPC. The same process
677
- creates and tracks only managed targets plus popup descendants with a verified
678
- managed opener chain. Its parent pipe is the liveness signal: daemon EOF,
679
- including `SIGKILL`, causes bounded descendant-first target cleanup. User
680
- targets are never adopted merely because they share a browser window. Browser
681
- disconnect clears that transient ownership before a new connection generation
682
- is exposed. No janitor method or raw target ID is part of the CLI or stdio
683
- protocol, and no ownership record is written to disk.
684
-
685
- ## Observation and Ref Semantics
686
-
687
- Observation v1 has these required public fields: `workspaceId`, `leaseId`,
688
- `targetId`, `url`, `observationId`, `title`, `elements`, `truncated`, and
689
- `truncationReasons`. Each element contains only `ref`, `role`, `name`, and the
690
- optional `value` and `checked` fields. Built-in Observation-producing tools
691
- also return `hints`; action tools additionally return discriminated `evidence`.
692
- The latter two fields remain optional in the schema so older compatible
693
- executors stay valid. Raw nodes, selectors, bounds, CDP identifiers, and the
694
- internal identity below never appear in an Observation result.
695
-
696
- The frozen v1 limits are 50 elements by default and 10,000 at most; 4,096
697
- string code units for title and element name; 16,384 for URL; 65,536 for one
698
- element value; 1,000,000 across collected Observation text; AX depth 128; and
699
- 2 MiB of UTF-8 serialized Observation data. Broker storage retains at most
700
- 2,048 Observations for five minutes each. Hitting any collection limit sets
701
- `truncated: true` and includes one or more canonical reasons in deterministic
702
- order: `element_limit`, `text_limit`, `depth_limit`, `byte_limit`.
703
-
704
- The public identity of an element is:
96
+ Browser Pilot itself does not ask for task-intent approval. The authorization
97
+ boundaries are:
705
98
 
706
- ```text
707
- workspaceId + observationId + ref
708
- ```
99
+ 1. The user enables Chrome remote debugging.
100
+ 2. Chrome presents its own connection authorization dialog when required.
101
+ 3. The Agent host decides whether the Agent may invoke local commands.
709
102
 
710
- Internal resolution includes at least:
103
+ A host may restrict operations through its own policy. Browser Pilot's default
104
+ surface remains complete and does not implement an approval state machine.
711
105
 
712
- ```text
713
- browserProcessIdentity
714
- browserConnectionGeneration
715
- targetId
716
- CDPSessionId
717
- frameId
718
- loaderId
719
- backendNodeId
720
- documentGeneration
721
- ```
106
+ ## 6. Browser Discovery and Connection
722
107
 
723
- The Broker stores process, connection, target, Lease, CDP session, frame,
724
- loader, and Document generation on the Observation; each ref adds its backend
725
- node identity. Resolution verifies all of them before live-node revalidation.
726
- Same-context failures expose one of these stable invalidation reasons when
727
- known: `navigation`, `loader_replaced`, `document_replaced`, `frame_changed`,
728
- `frame_detached`, `session_replaced`, `target_detached`,
729
- `browser_reconnected`, `target_ineligible`, `target_closed`,
730
- `control_released`, or `expired`.
731
- Ownership mismatches remain an undifferentiated `stale_ref` so they do not
732
- disclose another Workspace's state.
733
-
734
- Adding `document_replaced` as an optional `stale_ref.context.reason` value is
735
- additive under protocol 1.0 and 1.1: clients branch on the stable error code,
736
- handle known reasons when useful, and treat unknown reasons as a request to
737
- rebuild current Observation state. It adds no operation or capability.
738
-
739
- - **BR-13:** Cross-Workspace, cross-target, cross-observation, and expired refs
740
- return `stale_ref`; they never fall back to another target.
741
- - **BR-14:** Navigation, loader or Document replacement, target detach, frame
742
- detach, CDP session replacement, and browser reconnect hard-invalidate
743
- affected refs.
744
- - **BR-15:** Same-document DOM changes require live node revalidation. They do
745
- not force blanket invalidation when the node remains resolvable and valid.
746
- - **BR-16:** Observations are immutable and bounded by element count, text size,
747
- depth, and byte limits, with explicit truncation metadata.
748
- - **BR-28:** Agent hints are advisory, deterministic, bounded, and based only on
749
- browser-verifiable signals. They never contain field values, credentials,
750
- private file paths, raw CDP identities, or framework-specific prompt state.
751
-
752
- ## Events
753
-
754
- The event taxonomy includes navigation, document changed, target attached,
755
- target detached, control acquired/released, popup, dialog, download, connection
756
- lost, connection restored, network request, network response, command status,
757
- observation invalidated, Lease expiry, and watchdog signals for stalled
758
- navigation, selected-frame detach, unhandled dialogs, and repeated observable
759
- no-progress actions.
760
-
761
- Relevant event payloads carry the same `hints` representation: main-document
762
- 403/429 responses use `access_blocked`; downloads use `download` with the public
763
- Artifact ID only after completion; and the threshold no-progress event uses
764
- `repeated_action`. Subresource failures do not produce a page-blocked hint.
765
-
766
- Every event contains an event ID, monotonic Workspace sequence, timestamp,
767
- Workspace ID, relevant Lease and target IDs, type, payload version, and
768
- sensitivity. `workspaces/create|get` returns an explicit current `eventCursor`.
769
- Consumers poll from their last fully processed cursor; a poll returns ordered
770
- events, `nextCursor`, and `hasMore`. The Broker may compact old events and must
771
- return `cursor_expired` with retained cursor bounds rather than silently skip
772
- them. A consumer that loses its cursor rebuilds tab and Observation state, then
773
- uses `workspaces/get` to establish a new baseline.
774
-
775
- JavaScript dialogs enter explicit pending state and emit opened/closed events.
776
- They remain paused until the user or an Agent issues `browser.dialogs.respond`;
777
- Browser Pilot never auto-accepts them.
778
-
779
- Watchdogs report browser state; they do not make task decisions. A navigation
780
- that never becomes interactive emits `watchdog.navigation_stalled` and leaves
781
- the dispatched command as `unknown_outcome`. Detaching the selected frame emits
782
- `watchdog.frame_detached`, clears the frame selection, and invalidates its
783
- Observation. A dialog still pending after the bounded threshold emits one
784
- `watchdog.dialog_unhandled` event but remains pending. Three consecutive actions
785
- with browser-observable mismatch or no effect emit one `watchdog.no_progress`
786
- event per streak, scoped by Lease and target. Unobservable coordinate/canvas
787
- actions do not increment the streak. Navigation, frame changes, verified
788
- progress, session replacement, Lease release, and Workspace cleanup reset the
789
- applicable transient state.
790
-
791
- Watchdog payloads contain only bounded recovery metadata and opaque public IDs;
792
- they never contain typed values, passwords, refs, raw CDP target/session/frame
793
- IDs, or dialog prompt responses. Consumers inspect fresh browser state before
794
- deciding whether to continue, retry, or ask the user.
795
-
796
- ## Artifacts
797
-
798
- Artifact descriptors contain an opaque ID, kind, MIME type, byte size, original
799
- filename or dimensions when applicable, sensitivity, creation time, expiry, and
800
- optional preview relationship. They do not expose internal storage paths as the
801
- primary identity.
802
-
803
- - **BR-17:** Broker storage directories are mode `0700`; files are mode `0600`
804
- on platforms that support POSIX permissions.
805
- - **BR-18:** Artifacts have per-item, per-Workspace, and global capacity limits,
806
- TTL cleanup, retain/release, and explicit export to a client-owned path.
807
- - **BR-19:** Binary bytes are not written as base64 to the stdio stream. The
808
- local adapter reads an authorized Artifact or asks the CLI to export it, then
809
- converts it to the Agent runtime's image/file content type.
810
- - **BR-20:** Large or full-page screenshots return a model-sized preview plus
811
- an optional original Artifact.
812
-
813
- An adapter authorizes a local upload by calling `artifacts/import` with an
814
- absolute path, Workspace, and active Lease. The Broker copies the file into
815
- protected storage as a `user_file` / `upload_input` Artifact while preserving
816
- its basename. `browser.upload` accepts only that Artifact kind and never treats
817
- a screenshot, PDF, download, or arbitrary Broker-internal path as upload
818
- authorization.
819
-
820
- Downloads are event-produced Artifacts rather than a separate path-returning
821
- tool. CDP download behavior is browser-context scoped even when invoked through
822
- an attached target session, so per-session download directories cannot provide
823
- isolation. For each browser connection generation and Chrome Profile context,
824
- the Broker enables `Browser.downloadProgress` once with
825
- `Browser.setDownloadBehavior({ behavior: "default", eventsEnabled: true })`.
826
- The `default` behavior preserves Chrome's configured download directory and
827
- does not redirect user downloads.
828
-
829
- `Page.downloadWillBegin` on an attached controlled target binds the private CDP
830
- GUID to that target's Workspace and Lease. The browser-level completed event
831
- supplies the absolute source `filePath`; the Broker accepts it only for a GUID
832
- already bound by the Page event, validates a regular file, and copies it into a
833
- Workspace-owned `download` Artifact before publishing completion. Browser-level
834
- events without a controlled Page owner are ignored. GUIDs and source paths never
835
- appear in protocol output.
836
-
837
- The Browser Pilot copy is subject to tracking, item, Workspace, and global
838
- Artifact bounds. Exceeding a bound rejects only Artifact creation: it must not
839
- cancel, move, truncate, or delete Chrome's original download. Lease, session,
840
- Workspace, Artifact, and Broker cleanup stop tracking and delete only protected
841
- Artifact copies. The original file in the user's Chrome download directory is
842
- always left intact. If browser-level download events are unavailable, Browser
843
- Pilot reports capture as unavailable without changing Chrome's download
844
- behavior.
845
-
846
- ## Browser Discovery and Setup
847
-
848
- Discovery returns every supported local browser candidate with product,
849
- channel, user-data root, process state, remote-debugging state, authorization state,
850
- and structured remediation. It does not silently select the first filesystem
851
- match when several viable instances exist.
852
-
853
- After connection, Profile discovery separately returns every live regular
854
- Profile context on that endpoint. Browser candidates and Profile contexts are
855
- different layers: selecting Chrome Stable does not silently select one of its
856
- simultaneously open Profiles.
857
-
858
- The client may select an instance explicitly or use a deterministic persisted
859
- preference. Setup that requires Chrome UI remains a user-visible action. Browser
860
- Pilot cannot bypass Chrome authorization and must not simulate consent.
861
-
862
- Browser Pilot does not add an access-approval page. Products that want an
863
- approval UX own it in the Agent host before exposing the tool. Browser Pilot
864
- still excludes its own setup/status pages and other internal targets from
865
- Agent inventory; the same-user CDP limitation in **CON-7** remains.
866
-
867
- ## Security and Sensitive Data
868
-
869
- - Password inputs, cookies, auth credentials, network bodies, uploads,
870
- downloads, page captures, and selected page text carry sensitivity metadata.
871
- - Tool schemas mark selected text and page/element values as `browser_data`,
872
- with `credential` added where a field may carry a password, cookie, header,
873
- request body, response body, prompt response, or arbitrary eval value.
874
- - Artifact descriptors and BrowserEvents carry runtime sensitivity because
875
- their classification depends on the produced object or event. Their runtime
876
- value takes precedence over a tool's possible schema classifications.
877
- - Secrets are accepted through protected stdin or structured machine input, not
878
- command-line arguments in recommended workflows.
879
- - Audit records store metadata and hashes where possible, not secret values.
880
- - Public errors do not include cookies, credentials, network bodies, DOM dumps,
881
- or raw CDP payloads.
882
- - The Broker socket/pipe, configuration, and Artifact storage are restricted to
883
- the OS user.
884
- - Browser Pilot documents that another same-user process may bypass it and use
885
- Chrome's debugging endpoint directly.
886
-
887
- ## Stable Error Model
888
-
889
- Errors contain a stable code, human message, retryability, affected object IDs,
890
- and optional structured remediation. Required codes include:
108
+ Discovery returns structured candidates with:
891
109
 
892
- ```text
893
- protocol_incompatible not_initialized capability_denied
894
- browser_not_found browser_not_authorized browser_disconnected
895
- broker_in_use
896
- profile_selection_required profile_context_stale profile_context_unavailable
897
- workspace_not_found lease_expired target_not_owned
898
- target_busy stale_ref command_cancelled
899
- action_not_verified command_expired unknown_outcome
900
- artifact_not_found
901
- artifact_expired cursor_expired result_too_large
902
- invalid_argument internal_error
903
- ```
110
+ - stable browser ID;
111
+ - product and channel;
112
+ - user-data root;
113
+ - process state;
114
+ - remote-debugging state;
115
+ - authorization state;
116
+ - aggregate setup state;
117
+ - structured remediation.
904
118
 
905
- Human CLI output may add prose. Machine clients branch only on codes and typed
906
- fields, never on message text.
907
-
908
- ## Compatibility and Versioning
909
-
910
- - Protocol versions are independent of executable versions.
911
- - Compatible embedded clients reuse the running Broker even when their host or
912
- bundled executable versions differ. The one-shot compatibility CLI requires
913
- the running Broker's exact executable version because its external JSON
914
- contract is versioned separately from the bridge protocol.
915
- - An incompatible executable never replaces a live Broker. A host either uses
916
- a compatible executable or deliberately supplies a distinct
917
- `BROWSER_PILOT_HOME`; Browser Pilot never creates a hidden version namespace.
918
- - Broker shutdown is compare-and-stop: it requires the current Broker process
919
- identity and matching executable installation identity, and refuses while
920
- any embedded Connection remains live.
921
- - Additive tool and field changes use capability negotiation and schema
922
- evolution. Breaking semantics require a protocol-major change.
923
- - Unknown response fields are ignored; unknown request fields are rejected only
924
- when they change safety or semantics.
925
- - Optional advisory output such as `hints` does not require a protocol-minor
926
- version or capability when it adds no operation and only derives bounded data
927
- already covered by an existing granted capability.
928
- - The current CLI JSON shapes remain supported until a documented major CLI
929
- release and migration period.
930
- - A conformance suite validates any embedded adapter against the same executable
931
- used by direct CLI users.
932
-
933
- ## Functional Requirements and Acceptance
934
-
935
- - **FR-1 Universal installation:** An Agent can install Browser Pilot globally
936
- or locally and use existing one-shot commands.
937
- - **AC-1:** Existing documented CLI workflows and fixture tests pass after
938
- Broker migration without requiring an SDK, MCP server, or extension.
939
- - **FR-2 Embedded integration:** A product can bundle the official executable,
940
- initialize `bridge --stdio`, discover tools, execute them, receive events,
941
- and release resources.
942
- - **AC-2:** A reference adapter integrates using only executable invocation,
943
- the published protocol, and Artifact files; it imports no Browser Pilot
944
- source module.
945
- - **FR-3 Multi-client isolation:** Concurrent clients have isolated control and
946
- transient state.
947
- - **AC-3:** Two clients can operate separate Pilot windows concurrently, and
948
- refs, active targets, frames, auth, rules, and close operations cannot cross
949
- Workspace boundaries.
950
- - **FR-4 User-browser control:** An Agent can list and control all eligible
951
- current and future tabs in the selected BrowserInstance, including tabs the
952
- user opened before invoking it.
953
- - **AC-4:** No access-request or grant step is required. Inventory combines
954
- ManagedTabSets and user tabs behind Workspace-scoped opaque IDs. A Host may
955
- remove operations at launch. Concurrent control reports `target_busy`,
956
- release leaves user tabs open, and bulk cleanup closes only managed tabs.
957
- - **FR-5 Crash behavior:** Client and Broker failures have deterministic
958
- outcomes.
959
- - **AC-5:** Expired Leases are reclaimed; commands lost after dispatch report
960
- `unknown_outcome`; reconnect invalidates prior observations.
961
- - **FR-6 Media delivery:** Agent products can deliver Browser Pilot screenshots
962
- to multimodal models.
963
- - **AC-6:** Capture returns a protected Artifact and preview metadata; a
964
- reference adapter converts it to native image content without base64 on
965
- stdout.
966
- - **FR-7 Version safety:** User-installed and product-bundled clients coexist.
967
- - **AC-7:** Compatible clients reuse the Broker; incompatible clients receive
968
- `protocol_incompatible` or use a deliberately isolated Broker without
969
- terminating live clients.
970
- - **FR-8 Browser reliability:** Browser capability improvements do not alter the
971
- public lifecycle and isolation contract.
972
- - **AC-8:** Observation and action conformance tests pass against the public
973
- tool interface, including frame, navigation, stale-ref, obstruction, and
974
- input readback cases.
975
- - **FR-9 Multi-Profile routing:** One endpoint can expose all eligible tabs from
976
- multiple live Chrome Profiles without ambiguous new-target placement.
977
- - **AC-9:** Existing tabs remain directly controllable; every tab result has a
978
- connection-scoped ProfileContext ID; ambiguous open returns
979
- `profile_selection_required`; explicit or inferred routing is verified; and
980
- reconnect requires relisting rather than reusing stale Profile IDs.
981
-
982
- ## Non-Functional Requirements
983
-
984
- - **NFR-1:** No protocol message or normal tool result exceeds negotiated size
985
- limits; truncation is explicit.
986
- - **NFR-2:** Broker restart invalidates transient Workspaces, Leases, target
987
- assignments, refs, and commands. Clients rebuild state explicitly without
988
- guessing or replaying mutations.
989
- - **NFR-3:** Every command and security-sensitive lifecycle transition emits an
990
- auditable event without storing secret payloads.
991
- - **NFR-4:** Target command ordering is deterministic under concurrent clients.
992
- - **NFR-5:** Resource cleanup is idempotent and bounded after normal release,
993
- client crash, browser exit, and Broker restart.
994
-
995
- ## Non-Goals
996
-
997
- - Hosting browsers in the cloud or providing captcha-solving infrastructure.
998
- - Controlling Firefox or WebKit in the current architecture.
999
- - Providing OS-level isolation between processes owned by the same user.
1000
- - Defining whether an Agent's requested browser action matches user intent;
1001
- Agent hosts own semantic approval and confirmation policy.
1002
- - Modeling an Agent's conversation, run, task, worker, or model provider.
1003
- - Replacing URL Preview features inside embedding products.
1004
- - Making Browser Pilot an MCP server or publishing language SDKs.
1005
-
1006
- ## Verification Strategy
1007
-
1008
- 1. Protocol unit tests validate framing, negotiation, errors, schemas,
1009
- idempotency, cancellation, and cursor behavior.
1010
- 2. Broker integration tests create concurrent fake clients and assert complete
1011
- tab inventory, target assignment, Lease cleanup, crash recovery, and version
1012
- coexistence.
1013
- 3. Existing Playwright fixtures remain regression tests for human CLI behavior.
1014
- 4. Capability tests cover DOM/AX fusion, frames/OOPIF, stale refs, obstruction,
1015
- action verification, input readback, and document changes.
1016
- 5. Adapter conformance tests run against a language-neutral reference adapter
1017
- and at least two example Agent runtimes. No example-specific behavior enters
1018
- Broker production code.
1019
- 6. Browser-control tests prove that all eligible existing and future tabs are
1020
- included without grants, launch-time denied operations fail before CDP
1021
- dispatch, opaque IDs cannot cross Workspaces, and bulk cleanup never closes
1022
- user tabs.
1023
- 7. Distribution tests pack and install the npm package through global and local
1024
- layouts, launch local npx and product-owned absolute paths, and exercise
1025
- version output, bridge initialization, private Broker startup, and cleanup.
1026
- Native verification additionally checks manifest hashes, signature state,
1027
- private child roles, and archive checksums on each target platform.
1028
- 8. Multi-Profile tests cover passive discovery, explicit and inferred
1029
- selection, multiple ManagedTabSets, direct-create fallback, janitor adoption,
1030
- concurrent Workspace routing, reconnect invalidation, and protocol 1.1
1031
- ambiguous-open failure. Real Chrome acceptance verifies at least two live
1032
- Profiles on one endpoint without reconnecting or requesting authorization
1033
- again.
1034
-
1035
- ## Implementation Boundary
1036
-
1037
- The target internal boundary is:
119
+ Discovery is passive. It must not open a TCP or WebSocket connection and must
120
+ not display Chrome's Allow dialog.
1038
121
 
1039
- ```text
1040
- CLI formatter -----------\
1041
- stdio bridge -------------+--> BrowserPilotService --> Broker domain actors
1042
- conformance harness ------/ | --> CDP connector
1043
- +--> Artifact/Event stores
1044
- ```
122
+ Only an explicit browser connection command requests authorization. Concurrent
123
+ clients share one in-flight connection attempt. A failed or dropped connection
124
+ is never retried by a periodic timer. The next explicit command may request
125
+ reconnection.
126
+
127
+ One supervised browser-level CDP WebSocket is shared per endpoint. It must not
128
+ be multiplied per CLI process, Agent, profile, or tab.
129
+
130
+ ## 7. Profiles
131
+
132
+ One browser endpoint may expose multiple live Chrome profile contexts. Browser
133
+ Pilot must:
134
+
135
+ - inventory contexts passively from live target state;
136
+ - return representative eligible tabs;
137
+ - use connection-generation-scoped opaque profile IDs;
138
+ - route new managed tabs only after an unambiguous selection;
139
+ - preserve access to existing eligible tabs regardless of profile selection;
140
+ - invalidate profile IDs on browser reconnect.
141
+
142
+ Account-aware identity is an explicit visible operation. Browser Pilot may
143
+ open one temporary `chrome://version` page in each unidentified context, match
144
+ the exact profile path to bounded Local State metadata, then close the page.
145
+ It must not infer account identity from profile directory order or names.
146
+
147
+ If identity is unavailable or ambiguous, the result uses a neutral label and
148
+ representative tabs. The Agent must not silently select the first profile.
149
+
150
+ ## 8. Agent State Model
151
+
152
+ The public concept is an Agent namespace selected by
153
+ `BROWSER_PILOT_CLIENT_KEY` or `--client-key`.
154
+
155
+ Internally, the Broker maintains a principal, logical workspace, renewable
156
+ control lease, target mappings, selected profile/tab/frame, observations,
157
+ auth/network configuration, downloads, artifacts, commands, and bounded event
158
+ journal. These internal records are not exposed as a host protocol.
159
+
160
+ Requirements:
161
+
162
+ - The same stable key resumes one Agent's state across CLI processes.
163
+ - Different keys cannot inspect or mutate one another's state.
164
+ - A key must not be generated per command.
165
+ - State lives in Broker memory, not global refs or active-tab files.
166
+ - Browser reconnect invalidates all old browser-generation-scoped addresses.
167
+ - Expiry and explicit disconnect reclaim transient state and managed targets.
168
+ - User-opened tabs remain open during cleanup.
169
+
170
+ Compatible CLI versions may establish separate internal client sessions while
171
+ sharing the same Agent principal and keyed state. Compatibility is determined
172
+ by protocol negotiation, not executable path equality.
173
+
174
+ ## 9. Target Model and Concurrency
175
+
176
+ Every Agent receives opaque controlled-target addresses. Raw CDP target IDs and
177
+ session IDs are internal.
178
+
179
+ The inventory records origin as:
180
+
181
+ - `managed`;
182
+ - `managed_popup`;
183
+ - `user_tab`.
184
+
185
+ Requirements:
186
+
187
+ - The same physical user tab may appear in several Agent inventories.
188
+ - At most one Agent may control a physical tab at a time.
189
+ - A busy target returns stable `target_busy`; another Agent cannot steal it.
190
+ - Commands for the same physical target execute serially.
191
+ - Commands for independent targets may run concurrently.
192
+ - Explicit release removes the controlling session before another Agent can
193
+ acquire the target.
194
+ - A tab becoming ineligible invalidates its mapping without closing the tab.
195
+ - Bulk close operates only on managed tabs.
196
+ - Explicit close may close a selected user tab.
197
+
198
+ Managed popup adoption requires a complete owned opener chain. Browser Pilot
199
+ must fail closed when popup ownership cannot be proven.
200
+
201
+ ## 10. Observation and Refs
202
+
203
+ An observation fuses bounded accessibility semantics with DOM layout and state.
204
+ It includes:
205
+
206
+ - numbered interactive refs;
207
+ - role, accessible name, bounded value, and checked state;
208
+ - visibility, enabled/editable state, and geometry;
209
+ - open Shadow DOM;
210
+ - selected frame context;
211
+ - readable text and page geometry where requested;
212
+ - truncation metadata.
213
+
214
+ Each ref is scoped internally by Agent state, browser generation, target, CDP
215
+ session, frame, document/loader identity, observation generation, and backing
216
+ node identity.
217
+
218
+ Refs become stale after navigation, document replacement, semantic mutation,
219
+ target detach, frame change, reconnect, state release, or a newer invalidating
220
+ observation. A stale ref returns `stale_ref`; it must never be silently rebound
221
+ to another element.
222
+
223
+ Observation output is bounded by element count, field sizes, tree depth, text
224
+ length, and serialized byte budget. Truncation is explicit.
225
+
226
+ ## 11. Actions and Verification
227
+
228
+ Browser Pilot supports semantic click, type, press, keyboard input, upload,
229
+ scroll, and dropdown selection, plus coordinate input for visual surfaces.
230
+
231
+ Before dispatch, an element action must validate:
232
+
233
+ - current ref identity;
234
+ - target and frame continuity;
235
+ - enabled/editable state;
236
+ - live geometry;
237
+ - hit point and obstruction;
238
+ - browser connection generation.
239
+
240
+ After dispatch, Browser Pilot returns fresh page state and typed evidence. It
241
+ checks applicable effects such as:
242
+
243
+ - focus changes;
244
+ - checked/selected/pressed/expanded state;
245
+ - input or contenteditable value length and exact readback;
246
+ - navigation or document replacement;
247
+ - dialog or popup creation;
248
+ - upload file count and filename;
249
+ - scroll position;
250
+ - dropdown selection.
251
+
252
+ A mismatch returns a typed verification result or stable error. Sensitive
253
+ input evidence records only bounded metadata and never echoes password content.
254
+
255
+ If the page changes during a compound action, remaining steps are cancelled.
256
+ For example, type-with-submit must not press Enter after the input document is
257
+ replaced.
258
+
259
+ ## 12. Page Representations
260
+
261
+ The Agent can choose the smallest representation needed:
262
+
263
+ - snapshot for semantic controls;
264
+ - readable text for articles, lists, and results;
265
+ - targeted text search for a phrase and nearby context;
266
+ - bounded DOM metadata for selectors and attributes;
267
+ - page and scroll geometry for long documents;
268
+ - annotated screenshots for visual position and overlap;
269
+ - JavaScript evaluation only as an escape hatch.
270
+
271
+ This adaptive model incorporates the strongest relevant browser-use ideas
272
+ without adopting browser-use's Agent runtime or Python API.
273
+
274
+ ## 13. Async Operations and Command Recovery
275
+
276
+ The CLI provides browser-visible waits for URL, text, selector, dialog,
277
+ download, and popup conditions. Waiting uses a caller-defined deadline and
278
+ bounded polling interval. Timeout returns stable `wait_timeout` and does not
279
+ assert anything about an underlying mutation.
280
+
281
+ Every browser command has an internal command record with status:
282
+
283
+ - `accepted`;
284
+ - `dispatched`;
285
+ - `completed`;
286
+ - `unknown_outcome`;
287
+ - `cancelled`;
288
+ - `expired`.
289
+
290
+ The CLI exposes bounded command list, lookup, and cancellation commands.
291
+ `--request-id` supplies stable intended-call identity. The Broker derives
292
+ deterministic command and idempotency keys so a lost CLI result can be recovered
293
+ without redispatching the same intended call.
294
+
295
+ Cancellation is deterministic before dispatch and best effort after dispatch.
296
+ A connection loss or cancellation after mutation dispatch produces
297
+ `unknown_outcome` unless success or failure is known. The Agent must inspect
298
+ current browser state before retrying.
299
+
300
+ CLI `SIGINT` and `SIGTERM` request cancellation for the active command but do
301
+ not claim rollback.
302
+
303
+ ## 14. Dialogs, Auth, Cookies, and Network
304
+
305
+ Dialogs are scoped to owning Agent state and remain pending until explicitly
306
+ accepted or dismissed. Browser Pilot never chooses a dialog response.
307
+
308
+ HTTP authentication credentials and network rules are scoped to the Agent
309
+ namespace. Credential values are not retained outside the active configuration
310
+ needed by Chrome.
311
+
312
+ Cookie reads are scoped to the current URL or explicit domain.
313
+
314
+ Network support includes bounded request journals, detail/body reads, blocking,
315
+ mocking, header overrides, rule listing/removal, and clearing. Sensitive fields
316
+ carry internal sensitivity metadata and bounded output.
317
+
318
+ ## 15. Files and Media
319
+
320
+ CLI file-producing commands return:
321
+
322
+ - absolute local path;
323
+ - MIME type;
324
+ - byte size;
325
+ - width and height for images when available;
326
+ - command-specific metadata such as annotation count.
327
+
328
+ Output resolution order is:
329
+
330
+ 1. explicit filename;
331
+ 2. `BROWSER_PILOT_OUTPUT_DIR`;
332
+ 3. current working directory.
333
+
334
+ The Broker stores media and downloads in private bounded temporary storage,
335
+ then exports them to a caller-authorized absolute path. Binary bytes are never
336
+ written into normal CLI JSON.
337
+
338
+ Uploads copy a caller-authorized source into private storage and release it
339
+ after browser assignment. Downloads copy Chrome's completed file. Browser Pilot
340
+ never moves, truncates, deletes, or cancels the user's original file.
341
+
342
+ Storage has per-item, per-Agent, and global quotas plus expiry. Paths are not
343
+ derived from external IDs, symbolic-link escapes are rejected, and file modes
344
+ are private.
345
+
346
+ ## 16. Downloads
347
+
348
+ Download tracking is attached only to controlled sessions and browser
349
+ connection generation. Completed browser events are attributed to the owning
350
+ Agent before copying.
351
+
352
+ Requirements:
353
+
354
+ - downloads from unrelated or unowned tabs are ignored;
355
+ - duplicate browser GUID ownership fails closed;
356
+ - quota failure affects only the Browser Pilot copy;
357
+ - releasing Agent state stops tracking but does not cancel Chrome's download;
358
+ - reconnect configures browser-level download events once per generation;
359
+ - completed downloads can be listed and exported later by the same Agent key.
360
+
361
+ ## 17. Error Contract
362
+
363
+ Non-TTY CLI failures emit one JSON object with:
364
+
365
+ - `ok: false`;
366
+ - stable `code`;
367
+ - human-readable `error`;
368
+ - `retryable`;
369
+ - optional bounded `context`;
370
+ - optional `remediation`;
371
+ - optional recovery `hint`.
372
+
373
+ Consumers branch on `code`, never English text. Stable codes cover input,
374
+ authorization, browser lifecycle, profile routing, target contention, stale
375
+ state, verification, waiting, cancellation, uncertainty, quota, and internal
376
+ failure.
377
+
378
+ `retryable: true` means a later call may be valid. It never means the same
379
+ mutation is safe to replay.
380
+
381
+ ## 18. Process and Shutdown Model
382
+
383
+ The public CLI is short-lived. The per-user Broker and managed-target
384
+ connection supervisor are private background roles.
385
+
386
+ Startup uses a private lock and locator. Simultaneous CLIs create exactly one
387
+ Broker. A live but unresponsive Broker is never silently replaced.
388
+
389
+ The supervisor is the sole owner of Browser Pilot-created target identities.
390
+ After Broker crash or abrupt exit, it closes only managed targets and eligible
391
+ managed popup descendants. It never persists target IDs to disk and never
392
+ closes user-opened tabs.
393
+
394
+ `bp disconnect` first releases the current Agent namespace. It may stop the
395
+ whole Broker only when:
396
+
397
+ 1. no active Agent state remains;
398
+ 2. the Broker process identity has not changed;
399
+ 3. the invoking executable version and content/path identity match the
400
+ installation that started the Broker.
401
+
402
+ A compatible CLI from another installation succeeds in releasing its state but
403
+ leaves the Broker running.
404
+
405
+ ## 19. Distribution
406
+
407
+ Supported distributions are:
408
+
409
+ - global npm install;
410
+ - local npm install or `npx --no-install`;
411
+ - product-bundled npm package with pinned Node runtime;
412
+ - self-contained native executable.
413
+
414
+ Native targets are Apple Silicon macOS, x64 Linux, and x64 Windows. Intel Mac
415
+ is intentionally unsupported.
1045
416
 
1046
- `BrowserPilotService` is an internal architectural boundary, not a promised
1047
- Native SDK. CLI and bridge code may depend on it. Third-party products may not.
417
+ Every release includes:
418
+
419
+ - versioned native archives;
420
+ - archive checksum sidecars;
421
+ - per-file checksums and licenses inside native archives;
422
+ - an Agent skill/plugin archive and checksum;
423
+ - a release index binding CLI version, protocol range, skill compatibility
424
+ range, tested version, supported platforms, asset sizes, and hashes.
425
+
426
+ CLI, skill, plugin manifests, compatibility metadata, and checksums must be
427
+ released from one synchronized root version.
428
+
429
+ ## 20. Security and Privacy Invariants
430
+
431
+ - No browser extension is used or required.
432
+ - Broker runtime directories and files are private to the OS user.
433
+ - Public results never expose raw CDP target/session/node IDs.
434
+ - Cross-Agent object access fails closed.
435
+ - User tabs are not closed by release, crash cleanup, or bulk close.
436
+ - Browser authorization is never requested by discovery or timers.
437
+ - Output is bounded and carries truncation metadata.
438
+ - Password and credential values are not echoed in verification evidence.
439
+ - File operations reject unsafe paths and symbolic-link escapes.
440
+ - Raw CDP is not a public Agent interface.
441
+
442
+ ## 21. Acceptance Gates
443
+
444
+ A release is acceptable only when all of the following hold:
445
+
446
+ - TypeScript build succeeds.
447
+ - All unit and Broker process tests pass.
448
+ - Isolated Chrome capability metrics do not regress.
449
+ - Local Playwright core, compatibility, and network suites pass.
450
+ - npm global, local, and product-bundled distributions pass black-box tests.
451
+ - Native self-contained verification passes on release platforms.
452
+ - CLI help exposes `--client-key`, `--request-id`, `--timeout`, status, wait,
453
+ command recovery, and file commands.
454
+ - No removed persistent adapter command is present.
455
+ - Two Agent keys remain isolated while sharing one Broker.
456
+ - Compatible CLI installations reuse a Broker.
457
+ - A non-owner CLI cannot stop another installation's Broker.
458
+ - One physical user tab is never controlled concurrently by two Agents.
459
+ - Browser reconnect invalidates old refs and target identities.
460
+ - Managed crash cleanup leaves user tabs untouched.
461
+ - Screenshot, PDF, download, upload, and saved network results use local files
462
+ with complete metadata.
463
+
464
+ Real-site canaries report drift separately and do not make deterministic release
465
+ gates depend on third-party uptime.
466
+
467
+ ## 22. Non-Goals
468
+
469
+ - General cross-browser automation outside Chromium CDP.
470
+ - A remote multi-tenant browser service.
471
+ - Browser-extension capabilities such as tab-group management.
472
+ - A model provider or Agent orchestration framework.
473
+ - Per-action user-intent approval inside Browser Pilot.
474
+ - A public SDK, MCP server, or alternate machine protocol.