browser-pilot-cli 0.1.6 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +193 -27
- package/dist/cli.js +3861 -850
- package/dist/daemon.js +13749 -145
- package/dist/managed-target-janitor.js +490 -0
- package/docs/architecture/browser-pilot-platform-spec.md +965 -0
- package/docs/integration/stdio-bridge.md +614 -0
- package/docs/integration/stdio-conformance.md +51 -0
- package/docs/plans/browser-capability-evolution.md +410 -0
- package/docs/plans/universal-agent-integration.md +337 -0
- package/package.json +39 -6
- package/scripts/run-stdio-conformance.mjs +698 -0
- package/dist/chunk-77SA2ZLI.js +0 -18
- package/dist/defuddle-bundle-B7XUXIMB.js +0 -7
|
@@ -0,0 +1,614 @@
|
|
|
1
|
+
# Stdio Bridge Integration Contract
|
|
2
|
+
|
|
3
|
+
Status: browser `tools/call`, scoped Observations, target inventory, Artifacts,
|
|
4
|
+
command recovery, event delivery, browser reconnect, and scoped downloads
|
|
5
|
+
implemented.
|
|
6
|
+
|
|
7
|
+
This document describes the Agent-neutral process boundary. Tenon, OpenClaw,
|
|
8
|
+
and other Agent hosts use the same executable and protocol. No consumer imports
|
|
9
|
+
Browser Pilot source files, links a Native SDK, installs an extension, or needs
|
|
10
|
+
an MCP transport.
|
|
11
|
+
|
|
12
|
+
## Process Launch
|
|
13
|
+
|
|
14
|
+
Resolve an exact Browser Pilot executable version from one of these locations:
|
|
15
|
+
|
|
16
|
+
1. The product's bundled dependency.
|
|
17
|
+
2. A project-local npm dependency.
|
|
18
|
+
3. A compatible global installation selected explicitly by the host.
|
|
19
|
+
|
|
20
|
+
Launch it directly, without a shell:
|
|
21
|
+
|
|
22
|
+
```text
|
|
23
|
+
browser-pilot bridge --stdio
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The three supported layouts are behaviorally equivalent:
|
|
27
|
+
|
|
28
|
+
- global npm: the host explicitly resolves the installed `browser-pilot` bin;
|
|
29
|
+
- local npm/npx: the host pins `browser-pilot-cli` and resolves its local bin or
|
|
30
|
+
absolute `dist/cli.js` path;
|
|
31
|
+
- native bundle: the host pins and launches the release archive's absolute
|
|
32
|
+
`browser-pilot` executable path.
|
|
33
|
+
|
|
34
|
+
The npm layout requires Node 18 or newer and starts the adjacent packaged
|
|
35
|
+
`daemon.js` and `managed-target-janitor.js` files as private processes. A native
|
|
36
|
+
release is a Node SEA executable with its runtime and dependencies embedded. It
|
|
37
|
+
respawns that same executable for private Broker and janitor roles; reserved
|
|
38
|
+
`--browser-pilot-internal=*` arguments are not a Host API. It never downloads a
|
|
39
|
+
runtime or falls back to a system Node installation.
|
|
40
|
+
|
|
41
|
+
Native archives contain `manifest.json`, `SHA256SUMS`, dependency/runtime
|
|
42
|
+
licenses, and an adjacent archive checksum. Verify the archive checksum before
|
|
43
|
+
unpacking and the manifest-listed file hashes before launch. The manifest's
|
|
44
|
+
`signature.kind` is evidence, not an aspirational channel label: `developer_id`
|
|
45
|
+
or `adhoc` on macOS, `authenticode` or `unsigned` on Windows, and `unsigned` on
|
|
46
|
+
Linux. Hosts that require a trusted publisher must reject ad-hoc or unsigned
|
|
47
|
+
artifacts themselves.
|
|
48
|
+
|
|
49
|
+
The host may add `--browser <id|product|channel>` to make its initial browser
|
|
50
|
+
preference explicit. Without it, Browser Pilot deterministically selects the
|
|
51
|
+
first candidate with a recorded debugging endpoint, then stable platform order.
|
|
52
|
+
The running Broker retains that
|
|
53
|
+
choice in memory; it does not write target or profile selection state to disk.
|
|
54
|
+
|
|
55
|
+
Create pipes for stdin, stdout, and stderr. Stdout is protocol-only. Treat each
|
|
56
|
+
stderr line as a diagnostic and never parse it as protocol data.
|
|
57
|
+
|
|
58
|
+
The executable owns Broker discovery and startup. Hosts must not read PID
|
|
59
|
+
files, locator metadata, Unix socket paths, or Windows named-pipe names. Two
|
|
60
|
+
products that launch concurrently serialize through the per-user startup lock,
|
|
61
|
+
then both reuse the winning compatible Broker. A dead locator is recovered on
|
|
62
|
+
the next launch. Broker startup and browser discovery are passive. Only an
|
|
63
|
+
explicit `browser.connect` tool call requests Chrome authorization, and
|
|
64
|
+
concurrent calls for the same browser share one in-flight WebSocket attempt. A
|
|
65
|
+
ready live process with an
|
|
66
|
+
unresponsive endpoint is not killed or replaced; launch returns
|
|
67
|
+
`browser_disconnected` with `restart_unresponsive_broker` remediation.
|
|
68
|
+
|
|
69
|
+
The default is one shared Broker per OS user. Protocol-compatible clients reuse
|
|
70
|
+
it even when different Agent products or product versions launched their
|
|
71
|
+
bridges. If a product must run an incompatible Browser Pilot version, set an
|
|
72
|
+
absolute, product-owned `BROWSER_PILOT_HOME` before process launch. This creates
|
|
73
|
+
a deliberately isolated locator, endpoint, Artifact store, and version history
|
|
74
|
+
on macOS, Linux, and Windows. Do not silently choose an isolated home after
|
|
75
|
+
`protocol_incompatible`: shared-browser coordination and cleanup semantics must
|
|
76
|
+
remain an explicit host decision.
|
|
77
|
+
|
|
78
|
+
## Framing
|
|
79
|
+
|
|
80
|
+
The transport is JSON-RPC 2.0 over newline-delimited UTF-8 JSON:
|
|
81
|
+
|
|
82
|
+
- one complete JSON object per line;
|
|
83
|
+
- ordinary requests are dispatched in input order;
|
|
84
|
+
- request IDs are non-empty strings or safe integers;
|
|
85
|
+
- notifications have no response;
|
|
86
|
+
- the default input limit is 1 MiB per line and the default result limit is
|
|
87
|
+
4 MiB; protocol 1.1 clients may negotiate lower per-Connection limits;
|
|
88
|
+
- invalid JSON, invalid UTF-8, invalid envelopes, and oversized input terminate
|
|
89
|
+
that bridge process after an error response with `id: null` when possible;
|
|
90
|
+
- output observes stream backpressure;
|
|
91
|
+
- at most 256 ordinary calls and 16 out-of-band control calls are pending per
|
|
92
|
+
bridge process; saturation returns retryable `result_too_large` without
|
|
93
|
+
dispatching the rejected call;
|
|
94
|
+
- EOF releases every Lease owned by that bridge Connection;
|
|
95
|
+
- `shutdown` exits only the bridge process, not the shared per-user daemon.
|
|
96
|
+
|
|
97
|
+
Hosts must not send concurrent writes that interleave bytes. They may pipeline
|
|
98
|
+
complete lines and must correlate responses by JSON-RPC ID. `commands/get`,
|
|
99
|
+
`commands/cancel`, and dialog list/respond calls may overtake a pending
|
|
100
|
+
`tools/call`, so those responses can arrive earlier. Dialog control needs this
|
|
101
|
+
exception because a browser dialog can pause the command that caused it.
|
|
102
|
+
|
|
103
|
+
## Initialization
|
|
104
|
+
|
|
105
|
+
`initialize` must be the first successful request on a bridge Connection:
|
|
106
|
+
|
|
107
|
+
```json
|
|
108
|
+
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"client":{"id":"com.example.agent","name":"Example Agent","version":"2.0.0","instanceId":"install:01J..."},"protocol":{"min":{"major":1,"minor":1},"max":{"major":1,"minor":1}},"requestedCapabilities":["browser.control","workspace.manage","observation.read","action.input","artifact.read","event.read"],"launchMode":"embedded","limits":{"maxMessageBytes":1048576,"maxResultBytes":4194304}}}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The response returns the selected protocol, supported and granted
|
|
112
|
+
capabilities, executable and service versions, a process-stable Broker
|
|
113
|
+
identity, a Connection ID, browser candidates, and negotiated limits. Each
|
|
114
|
+
candidate has a stable `id`, product/channel/profile identity, aggregate
|
|
115
|
+
`state`, separate `processState`, `remoteDebuggingState`, and
|
|
116
|
+
`authorizationState`, plus structured `remediation` when action is needed.
|
|
117
|
+
Initialization succeeds when no candidate is ready, so a host can present setup
|
|
118
|
+
state and poll `browser.discover`. Discovery is passive and reports
|
|
119
|
+
authorization as `unknown` until an explicit connection succeeds or fails.
|
|
120
|
+
Workspace and Lease creation also remain passive; call `browser.connect` once
|
|
121
|
+
before target tools. Branch on structured `error.data.code`; never branch on
|
|
122
|
+
English error messages.
|
|
123
|
+
|
|
124
|
+
Protocol 1.0 uses the service limits returned by `initialize`. Protocol 1.1 may
|
|
125
|
+
send `limits.maxMessageBytes` and `limits.maxResultBytes`; each must be from
|
|
126
|
+
64 KiB through 1 GiB. The selected value is the smaller client/service maximum
|
|
127
|
+
and applies only to that bridge Connection. `maxArtifactBytes` and
|
|
128
|
+
`eventJournalSize` remain service resource limits, not client preferences.
|
|
129
|
+
|
|
130
|
+
The initialize request and response use the service's fixed bootstrap limits.
|
|
131
|
+
The bridge switches limits only after the successful response has been written,
|
|
132
|
+
and waits for that switch before parsing a pipelined next line. Oversized
|
|
133
|
+
responses become `result_too_large`; oversized best-effort notifications are
|
|
134
|
+
dropped and remain recoverable through `events/poll`.
|
|
135
|
+
|
|
136
|
+
An already-running incompatible daemon is never replaced. Initialization
|
|
137
|
+
returns `protocol_incompatible` with remediation to use a compatible executable
|
|
138
|
+
or a deliberate `BROWSER_PILOT_HOME` isolation boundary. Executable versions
|
|
139
|
+
and protocol versions are independent; successful protocol negotiation is what
|
|
140
|
+
allows embedded clients from different products or releases to share a Broker.
|
|
141
|
+
|
|
142
|
+
## Lifecycle Methods
|
|
143
|
+
|
|
144
|
+
The implemented lifecycle surface is:
|
|
145
|
+
|
|
146
|
+
```text
|
|
147
|
+
initialize
|
|
148
|
+
tools/list
|
|
149
|
+
tools/call
|
|
150
|
+
commands/get
|
|
151
|
+
commands/cancel
|
|
152
|
+
workspaces/create
|
|
153
|
+
workspaces/get
|
|
154
|
+
workspaces/release
|
|
155
|
+
leases/create
|
|
156
|
+
leases/heartbeat
|
|
157
|
+
leases/release
|
|
158
|
+
events/poll
|
|
159
|
+
artifacts/get
|
|
160
|
+
artifacts/export
|
|
161
|
+
artifacts/import
|
|
162
|
+
artifacts/retain
|
|
163
|
+
artifacts/release
|
|
164
|
+
shutdown
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
`workspaces/create` accepts an optional `browserId`. Protocol 1.1 clients may
|
|
168
|
+
also provide `clientKey` to make active Workspace creation idempotent within
|
|
169
|
+
their Principal; reusing a key for another browser fails. Without a browser ID,
|
|
170
|
+
the Broker prefers its first ready connection and otherwise uses its selected
|
|
171
|
+
stable-order browser binding. It returns a Workspace, its default logical
|
|
172
|
+
ManagedTabSet, and an `eventCursor`. `workspaces/get` returns the current cursor
|
|
173
|
+
as a recovery baseline. Creating a Workspace does not itself create a browser
|
|
174
|
+
connection or window. Call the Workspace-scoped `browser.connect` tool to
|
|
175
|
+
request Chrome authorization; the first managed navigation then creates the
|
|
176
|
+
dedicated browser window.
|
|
177
|
+
|
|
178
|
+
`leases/create` accepts a `workspaceId` and optional `ttlMs`. Protocol 1.1
|
|
179
|
+
clients may also provide a `clientKey`; repeating that key on the same live
|
|
180
|
+
Connection and Workspace returns the same Lease and renews its TTL. The default
|
|
181
|
+
Lease is 30 seconds, with a supported range of 1 second through 5 minutes. Heartbeat
|
|
182
|
+
well before `expiresAt`. A Lease belongs to its live Connection and cannot be
|
|
183
|
+
continued by a replacement bridge process. A replacement Connection from the
|
|
184
|
+
same ClientPrincipal may obtain a new Lease for a still-active Workspace.
|
|
185
|
+
|
|
186
|
+
Workspace identity belongs to `client.id + client.instanceId` for the lifetime
|
|
187
|
+
of the daemon. It is transient and is never restored from disk after Broker
|
|
188
|
+
restart. A Workspace released twice returns the same successful release result
|
|
189
|
+
while its bounded tombstone remains available.
|
|
190
|
+
|
|
191
|
+
## Tool Calls
|
|
192
|
+
|
|
193
|
+
`tools/list` returns only operations implemented by the running Broker and
|
|
194
|
+
allowed by negotiated capabilities. Call those tools with one uniform envelope:
|
|
195
|
+
|
|
196
|
+
```json
|
|
197
|
+
{"jsonrpc":"2.0","id":8,"method":"tools/call","params":{"name":"browser.observe","arguments":{"limit":50},"workspaceId":"workspace:...","leaseId":"lease:...","targetId":"target:...","commandId":"command:...","idempotencyKey":"observe:01J...","deadlineMs":30000}}
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
Connection tools omit Workspace fields. Workspace tools require
|
|
201
|
+
`workspaceId + leaseId`; target tools additionally require the opaque
|
|
202
|
+
`targetId` returned by `browser.tabs.list` or `browser.open`. Raw CDP target and
|
|
203
|
+
session IDs are never public inputs. The Broker validates the negotiated
|
|
204
|
+
capability, Lease ownership, tool schema, and target control before dispatch.
|
|
205
|
+
|
|
206
|
+
Each tool's `sensitivity.input` and `sensitivity.output` arrays list the classes
|
|
207
|
+
that may occur anywhere in that direction. Specific content-bearing schema
|
|
208
|
+
nodes carry `x-browser-pilot-sensitivity`, for example:
|
|
209
|
+
|
|
210
|
+
```json
|
|
211
|
+
{"type":"string","maxLength":1000000,"x-browser-pilot-sensitivity":["browser_data"]}
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
Use the field annotation when mapping `browser.read.result.text`, Observation
|
|
215
|
+
element values, auth fields, cookie values, network headers/bodies, upload
|
|
216
|
+
Artifact references, prompt text, or eval values into the Agent runtime. Keep
|
|
217
|
+
the annotation attached to derived model content; do not infer sensitivity from
|
|
218
|
+
field names or English tool descriptions. The annotation does not wrap or alter
|
|
219
|
+
the value. Artifact descriptors and BrowserEvents provide their actual runtime
|
|
220
|
+
`sensitivity` and should be propagated directly.
|
|
221
|
+
|
|
222
|
+
Every `tools/call` response is a Command outcome:
|
|
223
|
+
|
|
224
|
+
```json
|
|
225
|
+
{"command":{"id":"command:...","status":"completed","method":"browser.observe","browserConnectionGeneration":3,"idempotencyKey":"observe:01J...","acceptedAt":1,"deadlineAt":30001,"dispatchedAt":2,"completedAt":20,"mutating":false},"result":{"workspaceId":"workspace:...","leaseId":"lease:...","targetId":"target:...","url":"https://example.com","observationId":"observation:...","title":"Example","elements":[],"truncated":false,"truncationReasons":[]}}
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
The manifest's output schema describes the inner `result`. Clients should pass
|
|
229
|
+
a unique `commandId` when they may need cancellation; otherwise the Broker
|
|
230
|
+
generates one. `idempotencyKey` is optional, but retries must reuse it or reuse
|
|
231
|
+
the same caller-supplied `commandId`. Reusing a key for different arguments is
|
|
232
|
+
rejected. A duplicate completed call returns the recorded result; a duplicate
|
|
233
|
+
that is still accepted or dispatched returns its current Command without
|
|
234
|
+
dispatching again.
|
|
235
|
+
|
|
236
|
+
`commands/get` and `commands/cancel` accept `commandId` plus `workspaceId` for
|
|
237
|
+
Workspace commands. They are authorized by Principal ownership rather than the
|
|
238
|
+
original Lease, so a replacement bridge from the same product instance can
|
|
239
|
+
inspect a known outcome. Cancellation while accepted prevents browser dispatch.
|
|
240
|
+
Cancellation after dispatch is best-effort and sets `cancellationRequested`;
|
|
241
|
+
it never claims rollback. A mutating command whose deadline elapses after
|
|
242
|
+
browser dispatch becomes `unknown_outcome` and is never automatically replayed.
|
|
243
|
+
Known tool failures are stored as a completed Command with a nested JSON-RPC
|
|
244
|
+
`error`; the original call also returns that error normally.
|
|
245
|
+
|
|
246
|
+
`browser.click` returns a new Observation plus bounded evidence. For example,
|
|
247
|
+
a checkbox may return:
|
|
248
|
+
|
|
249
|
+
```json
|
|
250
|
+
{"action":"click","status":"verified","kind":"checkbox","effects":["checked_changed","focus_changed"],"checked":true,"focused":true}
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
Effects can also include `selected_changed`, `pressed_changed`,
|
|
254
|
+
`expanded_changed`, `navigation`, `document_changed`, `dialog_opened`, and
|
|
255
|
+
`popup_opened`. A semantic control that does not reach its expected state uses
|
|
256
|
+
`mismatch`; coordinate clicks and ref clicks with no conclusive observable
|
|
257
|
+
effect use `unavailable`. Treat evidence as browser-level execution evidence,
|
|
258
|
+
then inspect the returned Observation to decide whether the task itself
|
|
259
|
+
succeeded.
|
|
260
|
+
|
|
261
|
+
Every Observation produced by the built-in Browser tools also includes a
|
|
262
|
+
bounded `hints` array. Each item is discriminated by `code` and includes
|
|
263
|
+
`source`, `confidence`, and a stable `recommendedAction`:
|
|
264
|
+
|
|
265
|
+
```json
|
|
266
|
+
{"hints":[{"code":"modal_overlay","source":"observation","confidence":"strong","recommendedAction":"resolve_overlay_first","blocking":true,"refs":[2,5]}]}
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
Current codes are `autocomplete`, `modal_overlay`, `filter_controls`,
|
|
270
|
+
`authentication_surface`, `access_blocked`, `download`, and
|
|
271
|
+
`repeated_action`. Observation hints may contain only refs from the returned
|
|
272
|
+
Observation. Authentication state is `present` on the first observed auth
|
|
273
|
+
surface, `entered` when a session moves onto one, and `left` when it moves off
|
|
274
|
+
one. A session or selected-frame change resets that transition baseline.
|
|
275
|
+
|
|
276
|
+
Hints are advisory browser evidence. Use the recommended action as the next
|
|
277
|
+
inspection strategy, not as permission, proof of task success, or a reason to
|
|
278
|
+
skip normal error handling. Unknown hint codes and unknown response fields must
|
|
279
|
+
be ignored. The field is additive for protocol 1.0 and 1.1 and introduces no
|
|
280
|
+
new capability; its data is covered by the tool or event's existing
|
|
281
|
+
sensitivity and granted capability.
|
|
282
|
+
|
|
283
|
+
### Observation representations
|
|
284
|
+
|
|
285
|
+
The browser tools expose complementary bounded representations rather than one
|
|
286
|
+
unbounded page dump:
|
|
287
|
+
|
|
288
|
+
- `browser.observe` creates semantic elements and scoped refs for actions.
|
|
289
|
+
- `browser.read` returns broad readable text from the page or one CSS region.
|
|
290
|
+
- `browser.search` returns visible text matches, nearby context, and viewport
|
|
291
|
+
geometry, including open Shadow DOM, with at most 200 results.
|
|
292
|
+
- `browser.elements.find` returns bounded safe metadata for a CSS query,
|
|
293
|
+
optionally across open Shadow DOM. It never exposes DOM/CDP handles and its
|
|
294
|
+
results are not action refs.
|
|
295
|
+
- `browser.capture` may draw Observation ref boxes for spatial reasoning without
|
|
296
|
+
injecting an overlay into the page.
|
|
297
|
+
|
|
298
|
+
Observations may include an additive `page` object containing viewport and
|
|
299
|
+
document dimensions, scroll coordinates, pixels remaining in each direction,
|
|
300
|
+
and horizontal/vertical scroll percentages. Hosts must tolerate its absence
|
|
301
|
+
when talking to an older compatible Broker.
|
|
302
|
+
|
|
303
|
+
`browser.scroll` moves the page, an Observation ref, a CSS-selected container,
|
|
304
|
+
or a visible text match, then returns a fresh Observation and typed evidence.
|
|
305
|
+
`browser.dropdown.options` enumerates native or currently exposed ARIA options.
|
|
306
|
+
`browser.dropdown.select` verifies native value changes or uses fresh
|
|
307
|
+
Observation refs for custom controls. These tools reuse `observation.read` and
|
|
308
|
+
`action.input`; they are additive manifest entries, not a protocol-version
|
|
309
|
+
fork. Discover them through `tools/list`.
|
|
310
|
+
|
|
311
|
+
`browser.type` and `browser.keyboard` also return bounded evidence after the
|
|
312
|
+
page has had time to apply a controlled-field update:
|
|
313
|
+
|
|
314
|
+
```json
|
|
315
|
+
{"action":"type","status":"verified","kind":"input","sensitive":false,"beforeLength":3,"expectedLength":7,"afterLength":7}
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
Use `verification: "require_exact"` when a mismatch must fail the command.
|
|
319
|
+
Native text controls and contenteditable receive Chrome editing events;
|
|
320
|
+
date/time/color/range controls use their native value setter and emit
|
|
321
|
+
`beforeinput` plus `input`, but not an early `change`. Framework rollback,
|
|
322
|
+
browser value sanitization, canceled `beforeinput`, and editor interception are
|
|
323
|
+
reported from the effective value after the action. Password evidence never
|
|
324
|
+
contains the value itself.
|
|
325
|
+
|
|
326
|
+
`browser.type` and `browser.keyboard` are guarded composite actions. Browser
|
|
327
|
+
Pilot pins the target/session/frame/loader/Document identity and, once input
|
|
328
|
+
focus is established, the deep active element. It checks them between
|
|
329
|
+
select-all, deletion, individual keyboard characters, and submit. If a change
|
|
330
|
+
is detected before the first browser mutation, the call fails with retryable
|
|
331
|
+
`action_not_verified`. If one or more steps already ran, it fails with
|
|
332
|
+
`unknown_outcome`, for example:
|
|
333
|
+
|
|
334
|
+
```json
|
|
335
|
+
{"code":"unknown_outcome","retryable":true,"context":{"action":"keyboard","step":"type_character:4","reason":"focus_changed","dispatchedSteps":4,"remainingStepsStopped":true}}
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
Reasons are `target_changed`, `session_changed`, `frame_changed`,
|
|
339
|
+
`loader_changed`, `document_changed`, or `focus_changed`. On either failure,
|
|
340
|
+
list or switch to the intended target if necessary, create a fresh Observation,
|
|
341
|
+
and decide from current page state whether to continue. Never replay the whole
|
|
342
|
+
mutation automatically after `unknown_outcome`.
|
|
343
|
+
|
|
344
|
+
`browser.press` returns `action: "press"` with bounded effects such as
|
|
345
|
+
`value_changed`, `checked_changed`, `selected_changed`, `focus_changed`,
|
|
346
|
+
`navigation`, `document_changed`, `dialog_opened`, or `popup_opened`. It returns
|
|
347
|
+
`unavailable` with `no_observable_effect` when the key dispatch completed but no
|
|
348
|
+
supported effect was visible.
|
|
349
|
+
|
|
350
|
+
`browser.upload` reads the file input after assignment. It returns `verified`
|
|
351
|
+
only when one file with the expected browser filename remains selected,
|
|
352
|
+
`mismatch` when the page cleared or replaced the selection, and `unavailable`
|
|
353
|
+
when the target could no longer be read. Evidence contains counts and a filename
|
|
354
|
+
match boolean, never the Broker storage path.
|
|
355
|
+
|
|
356
|
+
Inventory includes every eligible ordinary user tab plus the Workspace's
|
|
357
|
+
managed tabs. A physical tab can be controlled by only one Lease at a time.
|
|
358
|
+
Commands for the same physical tab share one actor even though each Workspace
|
|
359
|
+
has a different opaque target ID. A controlling Lease can call
|
|
360
|
+
`browser.tabs.release` for one target without closing it. After that command
|
|
361
|
+
completes and emits `target_control.released`, another Lease may use its own
|
|
362
|
+
opaque ID to acquire the tab. This two-step handoff never accepts a destination
|
|
363
|
+
Lease ID and never silently steals control. Releasing a Workspace closes managed
|
|
364
|
+
tabs but leaves user tabs open.
|
|
365
|
+
|
|
366
|
+
## Events
|
|
367
|
+
|
|
368
|
+
Every event is first committed to a bounded per-Workspace journal and then
|
|
369
|
+
offered as this best-effort notification:
|
|
370
|
+
|
|
371
|
+
```json
|
|
372
|
+
{"jsonrpc":"2.0","method":"events/event","params":{"event":{"id":"event:...","sequence":7,"timestamp":1,"workspaceId":"workspace:...","browserConnectionGeneration":3,"leaseId":"lease:...","targetId":"target:...","type":"dialog","payloadVersion":1,"sensitivity":"browser_data","payload":{"dialogId":"dialog:...","state":"opened","type":"confirm","message":"Continue?","url":"https://example.com"}}}}
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
Notifications can interleave with responses and can be dropped under transport
|
|
376
|
+
backpressure or a disconnected bridge. Treat them as a low-latency signal, not
|
|
377
|
+
as the recovery record. Track the last fully processed cursor and poll:
|
|
378
|
+
|
|
379
|
+
```json
|
|
380
|
+
{"jsonrpc":"2.0","id":20,"method":"events/poll","params":{"workspaceId":"workspace:...","cursor":"cursor:6","limit":100}}
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
The result contains ordered `events`, `nextCursor`, and `hasMore`. Continue until
|
|
384
|
+
`hasMore` is false. If the cursor predates the retained journal, the Broker
|
|
385
|
+
returns `cursor_expired` with `earliestCursor` and `latestCursor`; rebuild tab
|
|
386
|
+
and observation state, then call `workspaces/get` to establish a new baseline.
|
|
387
|
+
Never infer delivery from notification arrival alone, and never advance the
|
|
388
|
+
stored cursor past an event the host has not processed.
|
|
389
|
+
|
|
390
|
+
Current producers cover command status, Lease expiry, navigation, document
|
|
391
|
+
change, target attach/detach, target control, managed popups, dialogs, downloads,
|
|
392
|
+
browser connection loss/restoration, network request/response metadata, and
|
|
393
|
+
observation invalidation. They also include `watchdog.navigation_stalled`,
|
|
394
|
+
`watchdog.frame_detached`, `watchdog.dialog_unhandled`, and
|
|
395
|
+
`watchdog.no_progress`. Network and watchdog events never contain credentials,
|
|
396
|
+
typed values, refs, headers, request/response bodies, or raw CDP IDs; retrieve
|
|
397
|
+
sensitive detail explicitly through the scoped tools.
|
|
398
|
+
|
|
399
|
+
`network.response` adds an `access_blocked` hint only for a main-document 403 or
|
|
400
|
+
429, not for failed images, scripts, XHR, or fetches. The threshold
|
|
401
|
+
`watchdog.no_progress` event and the action result that reaches the threshold
|
|
402
|
+
both contain the same `repeated_action` hint. The reason is either repeated
|
|
403
|
+
observable failure or `stagnant_page`, where the same action signature was
|
|
404
|
+
repeated while the bounded page fingerprint stayed unchanged. Fingerprints and
|
|
405
|
+
action parameters never leave the Broker. Change representation and strategy
|
|
406
|
+
instead of automatically repeating the same action or navigation.
|
|
407
|
+
|
|
408
|
+
`connection.lost` keeps the last connection generation and causes later browser
|
|
409
|
+
tools to fail with retryable `browser_disconnected`. The daemon passively
|
|
410
|
+
refreshes the originally selected profile's endpoint metadata and does not
|
|
411
|
+
switch profiles or open a WebSocket. The host calls `browser.connect` when it is
|
|
412
|
+
ready to request reconnection. `connection.restored` then carries a strictly
|
|
413
|
+
newer generation. Existing Workspaces
|
|
414
|
+
and active Leases remain valid, but old target IDs, frame IDs, CDP sessions,
|
|
415
|
+
Observations, and refs do not. Poll through the restoration event, call
|
|
416
|
+
`browser.tabs.list`, and rebuild target state. Never retry a mutating command whose
|
|
417
|
+
recorded status is `unknown_outcome`.
|
|
418
|
+
|
|
419
|
+
Every event has a top-level `browserConnectionGeneration`. A delayed ordinary
|
|
420
|
+
event from an older connection is discarded before journaling. Historical
|
|
421
|
+
`command.status` events and reconnect cleanup events remain in the journal with
|
|
422
|
+
their original generation, so consumers can finish old Command state machines
|
|
423
|
+
without treating cleanup as current browser state. Commands accepted against an
|
|
424
|
+
old generation are fenced before dispatch; commands already dispatched when the
|
|
425
|
+
connection changes follow the `unknown_outcome` rule above.
|
|
426
|
+
|
|
427
|
+
JavaScript dialogs remain pending. Use `browser.dialogs.list`, then call
|
|
428
|
+
`browser.dialogs.respond` with the returned `dialogId`, target, and explicit
|
|
429
|
+
`accept` or `dismiss` action. Browser Pilot never auto-accepts a dialog.
|
|
430
|
+
|
|
431
|
+
Watchdog events are advisory browser-state signals, not autonomous recovery:
|
|
432
|
+
|
|
433
|
+
- `watchdog.navigation_stalled` means `Page.navigate` was dispatched but the
|
|
434
|
+
page did not become interactive within 30 seconds. The command is
|
|
435
|
+
`unknown_outcome`; observe the target before deciding whether to navigate
|
|
436
|
+
again.
|
|
437
|
+
- `watchdog.frame_detached` carries the opaque selected `frameId`. Browser Pilot
|
|
438
|
+
clears that selection and invalidates the Observation; list frames and create
|
|
439
|
+
a fresh Observation before continuing.
|
|
440
|
+
- `watchdog.dialog_unhandled` is emitted once when a dialog remains pending for
|
|
441
|
+
15 seconds. It does not accept or dismiss the dialog.
|
|
442
|
+
- `watchdog.no_progress` is emitted once after three consecutive completed
|
|
443
|
+
actions on the same Lease and target either have browser-observable mismatch
|
|
444
|
+
or repeat the same action while the bounded page state remains stagnant.
|
|
445
|
+
Observable success resets the mismatch detector; a changed action or page
|
|
446
|
+
state resets the repetition detector. Navigation, frame/session changes, and
|
|
447
|
+
cleanup reset both. Coordinate/canvas actions whose effects cannot be observed
|
|
448
|
+
do not count as mismatch-only failures.
|
|
449
|
+
|
|
450
|
+
Browser Pilot never automatically retries, repeats, stops, accepts, or dismisses
|
|
451
|
+
an action because of a watchdog event. The host or Agent must inspect current
|
|
452
|
+
state and decide. Timers and streaks are in-memory transient state and are
|
|
453
|
+
removed on dialog close, session loss, Lease release, Workspace release, or
|
|
454
|
+
browser disconnect as applicable.
|
|
455
|
+
|
|
456
|
+
## Artifacts
|
|
457
|
+
|
|
458
|
+
`browser.capture` and `browser.pdf` return Artifact descriptors, never base64 or
|
|
459
|
+
an internal path. Use `artifacts/get` with the owning active Workspace and Lease
|
|
460
|
+
to obtain a protected local path, or `artifacts/export` with an absolute path to
|
|
461
|
+
copy the file to client-owned storage. Export does not overwrite by default.
|
|
462
|
+
|
|
463
|
+
Artifacts expire after 15 minutes by default. `artifacts/retain` extends that
|
|
464
|
+
to the retained TTL; `artifacts/release` removes the bytes immediately.
|
|
465
|
+
Workspace release and Broker shutdown also remove owned temporary bytes.
|
|
466
|
+
Directories use mode `0700` and files use `0600` where POSIX permissions apply.
|
|
467
|
+
|
|
468
|
+
Large screenshots default to a model-sized preview. Pass
|
|
469
|
+
`includeOriginal: true` to receive both the original descriptor and a preview
|
|
470
|
+
descriptor whose `previewOf` points to the original. The adapter reads the
|
|
471
|
+
selected file and converts it to its Agent runtime's native image/file content.
|
|
472
|
+
|
|
473
|
+
For spatial reasoning, a viewport-only `browser.capture` call may provide:
|
|
474
|
+
|
|
475
|
+
```json
|
|
476
|
+
{"annotations":{"observationId":"observation:...","refs":[1,3,8]}}
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
Omitting `refs` requests up to the first 200 refs in that Observation. The
|
|
480
|
+
Broker revalidates each ref, clips live geometry to the viewport, draws on an
|
|
481
|
+
unattached canvas in a Chrome isolated world, and returns `annotationCount`.
|
|
482
|
+
Screenshot bytes are not evaluated in the page's default world. Annotation
|
|
483
|
+
cannot be combined with `fullPage` or `selector`. A page can still move itself
|
|
484
|
+
between geometry sampling and capture, so hosts should use a recent Observation
|
|
485
|
+
and treat the image as visual context rather than durable element identity.
|
|
486
|
+
|
|
487
|
+
For upload, first call `artifacts/import` with the owning Workspace, active
|
|
488
|
+
Lease, and an absolute client-authorized path:
|
|
489
|
+
|
|
490
|
+
```json
|
|
491
|
+
{"jsonrpc":"2.0","id":30,"method":"artifacts/import","params":{"workspaceId":"workspace:...","leaseId":"lease:...","path":"/absolute/path/resume.pdf","mimeType":"application/pdf"}}
|
|
492
|
+
```
|
|
493
|
+
|
|
494
|
+
The Broker copies the source into protected storage and returns an
|
|
495
|
+
`upload_input` Artifact descriptor. Pass only its `artifactId` to
|
|
496
|
+
`browser.upload`; optionally identify a file input with
|
|
497
|
+
`observationId + ref` or `inputIndex`. Output Artifacts such as screenshots and
|
|
498
|
+
downloads are rejected as upload inputs. Workspace cleanup deletes the imported
|
|
499
|
+
copy but never deletes the client-owned source file.
|
|
500
|
+
|
|
501
|
+
Downloads have no path-returning tool. Once a controlled target session is
|
|
502
|
+
attached, the Broker configures target-session download capture and emits
|
|
503
|
+
`download` events with one of these states:
|
|
504
|
+
|
|
505
|
+
- `capture_unavailable`: that Chrome version rejected target-session isolation;
|
|
506
|
+
- `started`: includes an opaque `downloadId`, bounded URL, and suggested filename;
|
|
507
|
+
- `completed`: includes the Workspace-owned `download` Artifact descriptor;
|
|
508
|
+
- `failed` or `cancelled`: includes a stable, bounded reason and byte metadata.
|
|
509
|
+
|
|
510
|
+
These four lifecycle states also carry a `download` hint. `started` recommends
|
|
511
|
+
waiting, `completed` recommends inspecting the Artifact and includes only its
|
|
512
|
+
public `artifactId`, and terminal failures recommend inspecting the bounded
|
|
513
|
+
failure reason. `capture_unavailable` is setup state and does not claim that a
|
|
514
|
+
download started.
|
|
515
|
+
|
|
516
|
+
The private CDP GUID and staging path never appear in protocol output. On
|
|
517
|
+
`completed`, use the descriptor's `id` with `artifacts/get` or `artifacts/export`
|
|
518
|
+
exactly as for screenshots and PDFs. Staging directories are separate per CDP
|
|
519
|
+
session, use mode `0700`, enforce concurrency and byte quotas, and are removed on
|
|
520
|
+
session, Lease, or Workspace release. The session setting is restored to
|
|
521
|
+
`default` before an attached session is retired. Browser Pilot never calls browser-wide
|
|
522
|
+
`Browser.setDownloadBehavior`; unsupported target-session capture remains
|
|
523
|
+
unavailable instead of redirecting unrelated user downloads.
|
|
524
|
+
|
|
525
|
+
## Cleanup
|
|
526
|
+
|
|
527
|
+
Normal EOF and `shutdown` release Connection-owned Leases immediately. A killed
|
|
528
|
+
bridge is recovered by Lease expiry; stale Connection records and inactive
|
|
529
|
+
Workspaces are also reclaimed by bounded daemon sweeps. Releasing a Workspace
|
|
530
|
+
will eventually close only its Broker-managed targets. It must never close a
|
|
531
|
+
user tab merely because a client disconnected or a Workspace expired.
|
|
532
|
+
|
|
533
|
+
The daemon also starts a private managed-target janitor that solely owns the
|
|
534
|
+
browser-level CDP WebSocket and proxies daemon CDP traffic over private IPC. It
|
|
535
|
+
creates and tracks only Broker-managed targets and their managed popup
|
|
536
|
+
descendants. Daemon EOF, including an ungraceful process exit, makes the janitor
|
|
537
|
+
perform bounded cleanup without consulting disk or grouping tabs by browser
|
|
538
|
+
window. This internal process is not a public SDK, tool, or transport; embedded
|
|
539
|
+
clients still launch only `browser-pilot bridge --stdio`.
|
|
540
|
+
|
|
541
|
+
The stdio `shutdown` method exits only that bridge and disconnects its Broker
|
|
542
|
+
Connection. The human `bp disconnect` command separately releases the one-shot
|
|
543
|
+
compatibility Workspace and requests daemon shutdown. The daemon accepts that
|
|
544
|
+
request only when the Broker process identity and executable installation
|
|
545
|
+
identity still match and `embeddedConnections` is zero. Otherwise it returns
|
|
546
|
+
`protocol_incompatible` or retryable `broker_in_use`; it never evicts an Agent
|
|
547
|
+
product to complete an upgrade.
|
|
548
|
+
|
|
549
|
+
Owner-only `broker-locator.json` contains current executable and protocol
|
|
550
|
+
metadata. Owner-only `broker-versions.json` retains at most current and previous
|
|
551
|
+
executable installation metadata. Neither file contains browser selection,
|
|
552
|
+
tabs, target IDs, Workspaces, Leases, refs, Artifacts, credentials, network
|
|
553
|
+
rules, or command results. Rollback selection and preservation of old binaries
|
|
554
|
+
belong to the installer or embedding product, not to the Broker.
|
|
555
|
+
|
|
556
|
+
The current lifecycle runtime uses finite limits for live Connections,
|
|
557
|
+
per-Principal Workspaces, per-Connection Leases, and retained terminal records.
|
|
558
|
+
It does not persist target mappings, refs, cookies, credentials, network bodies,
|
|
559
|
+
or command state.
|
|
560
|
+
|
|
561
|
+
## Reference Consumer Adapters
|
|
562
|
+
|
|
563
|
+
The source repository includes
|
|
564
|
+
[`examples/adapters`](https://github.com/relixiaobo/browser-pilot/tree/main/examples/adapters)
|
|
565
|
+
implementations for a
|
|
566
|
+
Pi-style Tenon tool runtime and an OpenClaw dispatcher tool. They are host-side
|
|
567
|
+
examples, not npm package files, a Native SDK, or an additional supported
|
|
568
|
+
protocol surface. Production consumers should copy or port the small adapter
|
|
569
|
+
pattern into their own integration and continue treating this document plus the
|
|
570
|
+
runtime manifest as authoritative.
|
|
571
|
+
|
|
572
|
+
The shared example launches an absolute executable path without a shell,
|
|
573
|
+
negotiates protocol 1.1, discovers tools at runtime, and maps a host work scope
|
|
574
|
+
to a Workspace plus its active invocation to a renewable Lease. It never stores
|
|
575
|
+
an active tab, frame, Observation, ref, credential, rule, Artifact, or event
|
|
576
|
+
cursor on disk. Host tool-call IDs become Command and idempotency identities;
|
|
577
|
+
transport loss after dispatch of a mutating operation is returned as
|
|
578
|
+
non-retryable `unknown_outcome`.
|
|
579
|
+
|
|
580
|
+
For event recovery, call `pollEvents`, completely process the returned ordered
|
|
581
|
+
events, and only then call `acknowledgeEvents(nextCursor)`. A second poll before
|
|
582
|
+
acknowledgement intentionally starts from the same last processed cursor. On
|
|
583
|
+
`cursor_expired`, first rebuild tab and Observation state, then call
|
|
584
|
+
`resetEventCursor` to use `workspaces/get` as the new live baseline. Best-effort
|
|
585
|
+
notifications never advance the cursor.
|
|
586
|
+
|
|
587
|
+
Image Artifacts are read from their protected path, converted to the host's
|
|
588
|
+
native image content, and released. PDF and other file results require an
|
|
589
|
+
absolute host-owned output directory; they are exported there before the
|
|
590
|
+
protected Artifact is released. Cleanup invalidates local host contexts even if
|
|
591
|
+
the explicit Broker release fails, so Lease expiry remains a bounded fallback
|
|
592
|
+
and stale tools cannot continue issuing calls.
|
|
593
|
+
|
|
594
|
+
## Current Release Gate
|
|
595
|
+
|
|
596
|
+
`tools/list` is generated from the canonical schemas used for argument and
|
|
597
|
+
result validation, and production filtering prevents unwired tools from being
|
|
598
|
+
advertised. The current bridge supports discovery, connect, open, all-tab
|
|
599
|
+
inventory, observe/read/search/find, scroll and dropdown actions, annotated
|
|
600
|
+
screenshots, core actions, scoped frames, explicit dialogs, cookies, auth,
|
|
601
|
+
network observation/rules, eval, PDF, protected upload import, scoped download
|
|
602
|
+
Artifacts, Artifact access, command recovery, browser reconnect, and event
|
|
603
|
+
replay. Broker startup serialization, executable/protocol negotiation,
|
|
604
|
+
live-client shutdown protection, bounded version history, and deliberate
|
|
605
|
+
version isolation are covered by process tests. Packed global npm, local
|
|
606
|
+
npm/npx, and product-bundled absolute-path launches are also exercised as
|
|
607
|
+
black-box installs. Native artifact verification covers manifest hashes,
|
|
608
|
+
version output, stdio startup, the private Broker role, protected disconnect,
|
|
609
|
+
and the private janitor role without attaching to the user's browser.
|
|
610
|
+
|
|
611
|
+
Before shipping an adapter, run the black-box
|
|
612
|
+
[stdio conformance suite](stdio-conformance.md) against the exact bundled launch
|
|
613
|
+
command. The suite exercises only the public NDJSON protocol and creates no
|
|
614
|
+
user-tab control state.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Stdio Conformance Suite
|
|
2
|
+
|
|
3
|
+
The stdio conformance suite verifies an embedded Browser Pilot executable as a
|
|
4
|
+
black box. It launches the exact command supplied by the integrator, exchanges
|
|
5
|
+
only newline-delimited JSON-RPC 2.0 over stdin/stdout, and imports no Browser
|
|
6
|
+
Pilot source or protocol modules.
|
|
7
|
+
|
|
8
|
+
Build and test the repository executable:
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
npm run build
|
|
12
|
+
node scripts/run-stdio-conformance.mjs
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
An installed package also exposes the same runner as
|
|
16
|
+
`browser-pilot-conformance`.
|
|
17
|
+
|
|
18
|
+
Test an executable or product-bundled adapter with its exact arguments. The
|
|
19
|
+
command after `--` is passed directly to the operating system without a shell:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
node scripts/run-stdio-conformance.mjs -- \
|
|
23
|
+
/absolute/path/to/browser-pilot bridge --stdio
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Use `--report <path>` to write the same JSON report printed to stdout, and
|
|
27
|
+
`--timeout-ms <value>` to change the per-operation timeout. A passed suite exits
|
|
28
|
+
with status 0, a conformance failure exits with status 1, and invalid runner
|
|
29
|
+
arguments exit with status 2.
|
|
30
|
+
|
|
31
|
+
The suite creates one transient Workspace, one renewable Lease, and one managed
|
|
32
|
+
`about:blank` target. It does not enumerate or control user tabs. It validates:
|
|
33
|
+
|
|
34
|
+
- initialization and capability negotiation;
|
|
35
|
+
- tool manifest discovery;
|
|
36
|
+
- Workspace, Lease, and heartbeat lifecycle;
|
|
37
|
+
- managed target creation and inventory;
|
|
38
|
+
- a bounded Observation;
|
|
39
|
+
- screenshot creation, protected Artifact access, and release;
|
|
40
|
+
- event cursor polling;
|
|
41
|
+
- explicit target, Lease, Workspace, and bridge cleanup.
|
|
42
|
+
|
|
43
|
+
Failures trigger best-effort cleanup of only the resources created by the
|
|
44
|
+
suite. The versioned report contains check outcomes, counts, timings, version
|
|
45
|
+
metadata, and a bounded error summary. It intentionally omits opaque resource
|
|
46
|
+
IDs, Artifact paths, browser content, and bridge stderr text.
|
|
47
|
+
|
|
48
|
+
The repository tests exercise the runner against an independent fake stdio
|
|
49
|
+
executable, so normal unit tests do not connect to Chrome. Running the suite
|
|
50
|
+
against the real executable requires a ready supported browser and may start or
|
|
51
|
+
reuse the per-user Browser Pilot daemon.
|