pi-browser-harness 0.8.3 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +50 -0
- package/README.md +242 -181
- package/package.json +1 -1
- package/skills/deep-research/SKILL.md +66 -0
- package/skills/pi-browser-harness/SKILL.md +23 -0
- package/src/cdp/discovery.ts +183 -37
- package/src/cdp/ownership.ts +19 -1
- package/src/cdp/session.ts +74 -0
- package/src/cdp/target-factory.ts +223 -0
- package/src/cdp/window-binding.ts +15 -0
- package/src/client.ts +114 -27
- package/src/daemon/protocol.ts +12 -2
- package/src/daemon/spawn.ts +46 -13
- package/src/deep-research.ts +41 -0
- package/src/domains/forms.ts +355 -0
- package/src/domains/isolated-tab.ts +120 -0
- package/src/domains/navigate.ts +21 -13
- package/src/domains/readpage/capture.ts +46 -0
- package/src/domains/readpage/read-page.ts +119 -0
- package/src/domains/readpage/readability.ts +89 -0
- package/src/domains/render.ts +64 -0
- package/src/domains/search/extract.ts +44 -0
- package/src/domains/search/google-serp.ts +114 -0
- package/src/domains/search/web-search.ts +116 -0
- package/src/domains/setup.ts +4 -2
- package/src/index.ts +33 -7
- package/src/profile/bind.ts +138 -0
- package/src/profile/browser-process.ts +371 -0
- package/src/profile/command.ts +168 -0
- package/src/profile/launch.ts +90 -0
- package/src/profile/list.ts +171 -0
- package/src/profile/paths.ts +166 -0
- package/src/profile/picker.ts +73 -0
- package/src/profile/store.ts +105 -0
- package/src/prompt.ts +92 -116
- package/src/registry.ts +6 -0
- package/src/setup.ts +28 -44
- package/src/state.ts +1 -0
package/README.md
CHANGED
|
@@ -2,326 +2,387 @@
|
|
|
2
2
|
|
|
3
3
|

|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
**Browser control for [pi](https://github.com/badlogic/pi-mono) agents, in the Chrome you're already using.**
|
|
6
|
+
|
|
7
|
+
Your profile, your logins, your cookies. The agent reads pages as an accessibility tree with stable
|
|
8
|
+
element handles, fills forms that React won't revert, captures network traffic and console errors,
|
|
9
|
+
searches the web, and drops to raw CDP when the tools run out. 38 tools over a single CDP connection —
|
|
10
|
+
no MCP hop, no second browser, no extra API keys.
|
|
6
11
|
|
|
7
12
|
---
|
|
8
13
|
|
|
9
|
-
##
|
|
10
|
-
|
|
11
|
-
|
|
|
12
|
-
|
|
13
|
-
| **
|
|
14
|
-
| **
|
|
15
|
-
| **
|
|
16
|
-
| **
|
|
17
|
-
| **
|
|
18
|
-
| **
|
|
19
|
-
| **
|
|
20
|
-
| **
|
|
21
|
-
| **
|
|
22
|
-
| **
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
14
|
+
## How it compares
|
|
15
|
+
|
|
16
|
+
| | **pi-browser-harness** | Playwright MCP | Puppeteer MCP | Stagehand |
|
|
17
|
+
|---|---|---|---|---|
|
|
18
|
+
| **Browser** | Your running Chrome — real profile, live sessions | Launches a clean instance | Launches a clean instance | Launches a clean instance |
|
|
19
|
+
| **Element targeting** | AX-tree refs (`[eN]`) that re-resolve at action time; coordinate and selector fallbacks | Selectors / AX locators | CSS selectors | LLM-inferred selectors |
|
|
20
|
+
| **Input dispatch** | Compositor-level CDP input — works through iframes, shadow DOM, cross-origin frames | DOM-level | DOM-level | DOM-level |
|
|
21
|
+
| **Framework-safe writes** | `browser_fill` writes via the native setter and fires bubbling events, so controlled inputs keep the value | Native fill | Manual | Native fill |
|
|
22
|
+
| **Transport** | Native pi extension, in-process | MCP / JSON-RPC round trip | MCP / JSON-RPC round trip | In-process + its own LLM calls |
|
|
23
|
+
| **Extra credentials** | None | None | None | Its own model API key |
|
|
24
|
+
| **Concurrency** | Reads run in parallel; mutations auto-serialize through a shared mutex | Sequential | Sequential | Sequential |
|
|
25
|
+
| **Isolation** | Dedicated Chrome window per session; refuses tabs it didn't open | N/A (own browser) | N/A | N/A |
|
|
26
|
+
| **Terminal output** | Inline screenshots and `Ctrl+O`-expandable tool results | Plain text | Plain text | Plain text |
|
|
27
|
+
| **Escape hatch** | `browser_run_script` — raw CDP + full Node in one call | — | — | — |
|
|
28
|
+
|
|
29
|
+
<sub>Reflects each project's default configuration. Corrections welcome via PR.</sub>
|
|
30
|
+
|
|
31
|
+
**The short version:** if you live in pi and want an agent driving the same Chrome you're already
|
|
32
|
+
signed into, this is the one that fits.
|
|
28
33
|
|
|
29
34
|
---
|
|
30
35
|
|
|
31
|
-
##
|
|
36
|
+
## Install
|
|
32
37
|
|
|
33
38
|
```bash
|
|
34
|
-
# 1. Install
|
|
35
39
|
pi install npm:pi-browser-harness
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Then enable remote debugging in your browser — either open `chrome://inspect/#remote-debugging`
|
|
43
|
+
(or `brave://inspect`, `edge://inspect`), tick **Discover network targets**, and click **Allow** —
|
|
44
|
+
or relaunch the browser with `--remote-debugging-port=9222`.
|
|
36
45
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
# tick "Discover network targets", click Allow.
|
|
40
|
-
# Or launch Chrome with --remote-debugging-port=9222
|
|
46
|
+
Finally, run `/browser-setup` in pi to connect. It's a one-time step — the connection survives across
|
|
47
|
+
pi sessions and reconnects on its own when Chrome restarts.
|
|
41
48
|
|
|
42
|
-
|
|
43
|
-
|
|
49
|
+
The first time you connect, you'll be asked which browser profile the agent should work in:
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
Select the browser profile the agent should use
|
|
53
|
+
vertexcover.io (aman@vertexcover.io)
|
|
54
|
+
✓ Aman (personal@gmail.com)
|
|
55
|
+
Harness (harness@gmail.com)
|
|
56
|
+
— Clear selection (use whichever window is focused) —
|
|
44
57
|
```
|
|
45
58
|
|
|
46
|
-
|
|
59
|
+
That choice is saved to `~/.pi/agent/browser-harness.json` and reused by every project until you
|
|
60
|
+
change it with `/browser-profile`. It decides which logins, cookies, and extensions the agent works
|
|
61
|
+
with, so pinning it keeps runs reproducible. Skip the prompt and the harness behaves as it always
|
|
62
|
+
did: tabs open in whichever profile currently has focus.
|
|
47
63
|
|
|
48
|
-
|
|
49
|
-
- Node.js ≥ 22
|
|
50
|
-
- Chrome / Chromium / Edge
|
|
64
|
+
**Requires:** pi (latest) · Node.js ≥ 22 · Chrome, Chromium, Brave, or Edge
|
|
51
65
|
|
|
52
66
|
---
|
|
53
67
|
|
|
54
|
-
##
|
|
68
|
+
## The mental model
|
|
69
|
+
|
|
70
|
+
Interaction is **ref-first**. `browser_snapshot` returns the accessibility tree, and every
|
|
71
|
+
interactive element carries a stable ref (`[eN]`) plus click coordinates `@(x,y)`:
|
|
55
72
|
|
|
56
73
|
```
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
│ → browser_snapshot (DEFAULT — AX tree with @(x,y) per interactive element)
|
|
61
|
-
│
|
|
62
|
-
├─ A specific element's value / attribute / coords?
|
|
63
|
-
│ → browser_execute_js (e.g. el.innerText, el.getBoundingClientRect())
|
|
64
|
-
│
|
|
65
|
-
├─ Network behavior on the current page?
|
|
66
|
-
│ → browser_network_requests
|
|
67
|
-
│
|
|
68
|
-
└─ Visual rendering (layout / colors / chart drew correctly)?
|
|
69
|
-
→ browser_screenshot (LAST RESORT — pixels only)
|
|
74
|
+
[e7] textbox "Email" @(412,288)
|
|
75
|
+
[e9] combobox "Country" @(412,344)
|
|
76
|
+
[e12] button "Create account" @(456,412)
|
|
70
77
|
```
|
|
71
78
|
|
|
72
|
-
Pass
|
|
79
|
+
Pass the ref — not the coordinates — to `browser_click`, `browser_fill`, `browser_select_option`,
|
|
80
|
+
`browser_focus`, `browser_upload_file`, or `browser_dispatch_key`. Refs are keyed to element
|
|
81
|
+
identity, so they re-resolve position at action time and survive the re-renders that SPA forms
|
|
82
|
+
trigger on every keystroke. Coordinates go stale the moment the layout reflows.
|
|
83
|
+
|
|
84
|
+
Three things follow from that:
|
|
85
|
+
|
|
86
|
+
- **Never screenshot to find a click target.** The snapshot already has refs and coordinates.
|
|
87
|
+
- **Mutating calls append a "Page changes" diff.** Read it to confirm the step landed — a panel
|
|
88
|
+
closed, a new `[eN]` appeared — instead of taking a screenshot to check.
|
|
89
|
+
- **`"ref is stale"` means the page changed.** Re-snapshot for fresh refs; don't retry blindly.
|
|
90
|
+
|
|
91
|
+
### Which tool answers your question
|
|
92
|
+
|
|
93
|
+
| You need to know… | Reach for |
|
|
94
|
+
|---|---|
|
|
95
|
+
| What's on the page, what's clickable | `browser_snapshot` — the default |
|
|
96
|
+
| One element's text, value, or geometry | `browser_execute_js` — one round trip |
|
|
97
|
+
| An article's readable content | `browser_read_page` |
|
|
98
|
+
| Why nothing happened after an action | `browser_console` |
|
|
99
|
+
| What the page requested over the network | `browser_network_requests` |
|
|
100
|
+
| Whether something *rendered* correctly | `browser_screenshot` — last resort, pixels only |
|
|
73
101
|
|
|
74
102
|
---
|
|
75
103
|
|
|
76
104
|
## Tools
|
|
77
105
|
|
|
78
|
-
###
|
|
106
|
+
### Reading the page
|
|
79
107
|
|
|
80
108
|
| Tool | Purpose |
|
|
81
|
-
|
|
82
|
-
| `browser_snapshot` | **Default
|
|
83
|
-
| `browser_execute_js` | Surgical DOM reads —
|
|
84
|
-
| `
|
|
85
|
-
| `
|
|
86
|
-
| `
|
|
109
|
+
|---|---|
|
|
110
|
+
| `browser_snapshot` | **Default.** Accessibility tree with `[eN]` refs and `@(x,y)` per interactive element. Optional `includeScreenshot`. |
|
|
111
|
+
| `browser_execute_js` | Surgical DOM reads — text, attributes, `getBoundingClientRect()`. Cheapest precise read. |
|
|
112
|
+
| `browser_page_info` | URL, title, viewport, scroll position — or the pending JS dialog, if one is blocking. |
|
|
113
|
+
| `browser_console` | JS errors and console output. Pass `sinceSeq` from the previous `nextCursor` to see only what an action caused. |
|
|
114
|
+
| `browser_screenshot` | PNG/JPEG capture. For verifying visual rendering only. |
|
|
87
115
|
|
|
88
|
-
###
|
|
116
|
+
### Web research
|
|
89
117
|
|
|
90
118
|
| Tool | Purpose |
|
|
91
|
-
|
|
92
|
-
| `
|
|
119
|
+
|---|---|
|
|
120
|
+
| `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real Google SERP in your Chrome. No API key. Links only. |
|
|
121
|
+
| `browser_read_page` | Reader mode. A `url` (opened and closed in its own tab) or an owned `targetId` → clean article text, boilerplate stripped. |
|
|
122
|
+
|
|
123
|
+
Both run in isolated tabs and never disturb your current page.
|
|
124
|
+
|
|
125
|
+
### Interacting
|
|
93
126
|
|
|
94
|
-
|
|
127
|
+
| Tool | Purpose |
|
|
128
|
+
|---|---|
|
|
129
|
+
| `browser_click` | Click by `ref` (preferred) or viewport `x`/`y`. Compositor-level, so iframes and shadow DOM work. |
|
|
130
|
+
| `browser_fill` | Set an input, textarea, or contenteditable. Fires bubbling `input`/`change`, returns the value written. |
|
|
131
|
+
| `browser_select_option` | Choose in a native `<select>` by value, label, or index. Returns available options on no match. |
|
|
132
|
+
| `browser_focus` | Deterministic `.focus()` — no coordinate accuracy needed. |
|
|
133
|
+
| `browser_type` | Type into the focused element. For keystroke-sensitive widgets; prefer `browser_fill` otherwise. |
|
|
134
|
+
| `browser_press_key` | Press a key with an optional modifier bitfield. |
|
|
135
|
+
| `browser_dispatch_key` | Synthetic DOM `KeyboardEvent` on one element, for pages that ignore raw CDP keys. |
|
|
136
|
+
| `browser_scroll` | Scroll by delta. W3C wheel convention: positive `deltaY` = down. |
|
|
137
|
+
| `browser_drag_and_drop` | Drag between two coordinate pairs. |
|
|
138
|
+
|
|
139
|
+
### Navigation and tabs
|
|
95
140
|
|
|
96
141
|
| Tool | Purpose |
|
|
97
|
-
|
|
98
|
-
| `browser_navigate`
|
|
99
|
-
| `browser_open_urls` | Open
|
|
100
|
-
| `browser_go_back` / `browser_go_forward` / `browser_reload` | History
|
|
101
|
-
| `browser_list_tabs` / `browser_current_tab`
|
|
142
|
+
|---|---|
|
|
143
|
+
| `browser_navigate` | Go to a URL in the current tab. |
|
|
144
|
+
| `browser_new_tab` / `browser_open_urls` | Open one tab, or many URLs in parallel. |
|
|
145
|
+
| `browser_go_back` / `browser_go_forward` / `browser_reload` | History. |
|
|
146
|
+
| `browser_list_tabs` / `browser_current_tab` | List tabs (`scope:"owned"` by default) and inspect the attached one. |
|
|
147
|
+
| `browser_switch_tab` / `browser_close_tab` | Switch or close — owned tabs only. |
|
|
102
148
|
|
|
103
|
-
###
|
|
149
|
+
### Waiting and dialogs
|
|
104
150
|
|
|
105
151
|
| Tool | Purpose |
|
|
106
|
-
|
|
107
|
-
| `
|
|
108
|
-
| `
|
|
109
|
-
| `
|
|
110
|
-
| `
|
|
152
|
+
|---|---|
|
|
153
|
+
| `browser_wait_for` | Wait until a selector appears (or disappears with `gone:true`), or text shows up. **Prefer this over sleeping.** |
|
|
154
|
+
| `browser_wait_for_load` | Wait for `document.readyState === "complete"`. |
|
|
155
|
+
| `browser_wait` | Fixed delay, in seconds. |
|
|
156
|
+
| `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt`. |
|
|
111
157
|
|
|
112
|
-
###
|
|
158
|
+
### Network
|
|
113
159
|
|
|
114
160
|
| Tool | Purpose |
|
|
115
|
-
|
|
116
|
-
| `
|
|
117
|
-
| `
|
|
118
|
-
| `browser_run_script` | Execute a temporary script with daemon and Node.js access |
|
|
161
|
+
|---|---|
|
|
162
|
+
| `browser_network_requests` | Requests captured on the current tab since attach. Filter by URL, method, status, resource type, or recency; optional response bodies. |
|
|
163
|
+
| `browser_http_get` | Plain GET outside the browser. Much faster than navigate-then-read for JSON APIs — but no JS rendering. |
|
|
119
164
|
|
|
120
|
-
|
|
165
|
+
### Files and viewport
|
|
121
166
|
|
|
122
|
-
|
|
167
|
+
| Tool | Purpose |
|
|
168
|
+
|---|---|
|
|
169
|
+
| `browser_upload_file` | Set files on a file input, with a DataTransfer fallback for stubborn pages. |
|
|
170
|
+
| `browser_download` | Set the download directory and suppress the save-as prompt. |
|
|
171
|
+
| `browser_print_to_pdf` | Print the page to PDF via `Page.printToPDF`. |
|
|
172
|
+
| `browser_viewport_resize` | Override viewport size and device pixel ratio for responsive checks. |
|
|
123
173
|
|
|
124
|
-
|
|
174
|
+
### Escape hatches
|
|
125
175
|
|
|
126
|
-
|
|
176
|
+
| Tool | Purpose |
|
|
177
|
+
|---|---|
|
|
178
|
+
| `browser_run_script` | Run a temporary script with raw CDP and full Node access. See [Extending](#extending). |
|
|
179
|
+
| `browser_setup` | Agent-callable setup. Idempotent — how the agent recovers from `not initialized` on its own. |
|
|
127
180
|
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
181
|
+
`browser_snapshot`, `browser_network_requests`, `browser_console`, `browser_execute_js`,
|
|
182
|
+
`browser_web_search`, and `browser_read_page` ship custom TUI rendering: compact by default,
|
|
183
|
+
full output on **`Ctrl+O`** (`app.tools.expand`). The model always receives the complete result
|
|
184
|
+
regardless of render state.
|
|
185
|
+
|
|
186
|
+
---
|
|
187
|
+
|
|
188
|
+
## Recipes
|
|
132
189
|
|
|
133
|
-
###
|
|
190
|
+
### Fill a form on an SPA
|
|
191
|
+
|
|
192
|
+
Wait, snapshot once, then act by ref. No screenshots anywhere in this loop.
|
|
134
193
|
|
|
135
194
|
```
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
browser_snapshot() # verify next state
|
|
195
|
+
browser_wait_for({ selector: "#email" })
|
|
196
|
+
browser_snapshot() // → [e7] textbox, [e9] combobox, [e12] button
|
|
197
|
+
browser_fill({ ref: "e7", value: "a@b.com" })
|
|
198
|
+
browser_select_option({ ref: "e9", label: "India" })
|
|
199
|
+
browser_click({ ref: "e12" }) // re-resolves position even after reflow
|
|
142
200
|
```
|
|
143
201
|
|
|
144
|
-
|
|
202
|
+
Each mutating call returns a "Page changes" diff — that's your confirmation the step landed.
|
|
203
|
+
|
|
204
|
+
### Extract data
|
|
145
205
|
|
|
146
206
|
```js
|
|
147
207
|
// One value
|
|
148
208
|
browser_execute_js({ expression: "document.querySelector('.price').innerText" })
|
|
149
209
|
|
|
150
|
-
//
|
|
151
|
-
browser_http_get({ url: "https://api.github.com/repos/amankumarsingh77/pi-browser-harness" })
|
|
152
|
-
|
|
153
|
-
// Structured arrays
|
|
210
|
+
// Structured
|
|
154
211
|
browser_execute_js({ expression: `JSON.stringify(
|
|
155
212
|
Array.from(document.querySelectorAll('.result')).map(el => ({
|
|
156
213
|
title: el.querySelector('h3')?.textContent,
|
|
157
214
|
link: el.querySelector('a')?.href,
|
|
158
215
|
}))
|
|
159
216
|
)` })
|
|
217
|
+
|
|
218
|
+
// Skip the browser entirely for a JSON endpoint
|
|
219
|
+
browser_http_get({ url: "https://api.github.com/repos/amankumarsingh77/pi-browser-harness" })
|
|
160
220
|
```
|
|
161
221
|
|
|
162
|
-
###
|
|
222
|
+
### Debug a failing page
|
|
163
223
|
|
|
164
224
|
```
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
browser_network_requests({
|
|
225
|
+
browser_click({ ref: "e12" })
|
|
226
|
+
browser_console({ levels: ["error", "warn"] }) // did JS throw?
|
|
227
|
+
browser_network_requests({ // did a request 4xx/5xx?
|
|
168
228
|
urlPattern: "/api/",
|
|
169
229
|
statusFilter: { min: 400 },
|
|
170
230
|
includeResponseBodies: true
|
|
171
231
|
})
|
|
172
232
|
```
|
|
173
233
|
|
|
174
|
-
###
|
|
234
|
+
### Search, then read
|
|
175
235
|
|
|
176
236
|
```
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
browser_list_tabs()
|
|
180
|
-
browser_switch_tab({ targetId: "..." })
|
|
181
|
-
browser_wait_for_load()
|
|
182
|
-
browser_snapshot()
|
|
183
|
-
browser_execute_js({ expression: "document.querySelector('.content').innerText" })
|
|
237
|
+
browser_web_search({ query: "chrome devtools protocol overview", limit: 5 })
|
|
238
|
+
browser_read_page({ url: "<the best result's url>" })
|
|
184
239
|
```
|
|
185
240
|
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
browser_screenshot() # ONLY if you need to verify a chart/modal/CSS rendered correctly
|
|
192
|
-
```
|
|
241
|
+
For a question that needs several sources synthesized into a **cited report**, use
|
|
242
|
+
`/deep-research <question>`. It decomposes the question, fans out one isolated
|
|
243
|
+
`web-search-researcher` subagent per sub-question, loops for coverage under a hard ceiling, and
|
|
244
|
+
writes a Markdown report where every claim carries a source link. The skill also auto-triggers on
|
|
245
|
+
research-shaped requests without the command.
|
|
193
246
|
|
|
194
247
|
### Keyboard modifiers
|
|
195
248
|
|
|
196
|
-
|
|
197
|
-
|-----|-----|
|
|
198
|
-
| Alt | 1 |
|
|
199
|
-
| Ctrl | 2 |
|
|
200
|
-
| Meta / Cmd | 4 |
|
|
201
|
-
| Shift | 8 |
|
|
249
|
+
Modifiers are a bitfield: `1` Alt · `2` Ctrl · `4` Meta/Cmd · `8` Shift.
|
|
202
250
|
|
|
203
251
|
```
|
|
204
252
|
browser_press_key({ key: "c", modifiers: 2 }) // Ctrl+C
|
|
205
|
-
browser_press_key({ key: "v", modifiers: 4 }) // Cmd+V
|
|
206
253
|
browser_press_key({ key: "T", modifiers: 10 }) // Ctrl+Shift+T
|
|
207
254
|
```
|
|
208
255
|
|
|
209
|
-
### Dialogs
|
|
210
|
-
|
|
211
|
-
JS dialogs freeze the page. Check `browser_page_info` first — if it reports a dialog, handle it before anything else:
|
|
212
|
-
|
|
213
|
-
```
|
|
214
|
-
browser_handle_dialog({ action: "accept" }) // confirm
|
|
215
|
-
browser_handle_dialog({ action: "dismiss" }) // cancel
|
|
216
|
-
browser_handle_dialog({ action: "accept", promptText: "hello" }) // prompt
|
|
217
|
-
```
|
|
218
|
-
|
|
219
256
|
---
|
|
220
257
|
|
|
221
|
-
##
|
|
258
|
+
## Concurrency and isolation
|
|
222
259
|
|
|
223
|
-
|
|
260
|
+
**Parallel reads, serialized writes.** Observation tools run concurrently with each other and with
|
|
261
|
+
mutations. Mutations (`click`, `fill`, `type`, `scroll`, `navigate`, `switch_tab`, …) pass through a
|
|
262
|
+
shared async mutex and FIFO-queue automatically, so emitting several in one turn is safe.
|
|
224
263
|
|
|
225
264
|
```
|
|
226
|
-
|
|
227
|
-
browser_snapshot()
|
|
228
|
-
browser_network_requests({ sinceMs: 5000 })
|
|
229
|
-
browser_http_get({ url: "..." })
|
|
265
|
+
browser_snapshot() + browser_network_requests({ sinceMs: 5000 }) + browser_http_get({ url: "..." })
|
|
230
266
|
```
|
|
231
267
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
The harness never touches tabs you didn't open through it. On first attach it spawns a dedicated Chrome window; subsequent `browser_new_tab` calls open inside that window. `browser_list_tabs` defaults to `scope:"owned"` (pass `scope:"all"` to see read-only listings of your other tabs); `browser_switch_tab` and `browser_close_tab` refuse non-owned tabs.
|
|
268
|
+
**Tab ownership.** On first attach the harness opens a dedicated Chrome window and only operates
|
|
269
|
+
inside it. `browser_list_tabs` defaults to `scope:"owned"` (pass `scope:"all"` for a read-only view
|
|
270
|
+
of your other tabs); `browser_switch_tab` and `browser_close_tab` refuse anything this session
|
|
271
|
+
didn't open. Owned tabs are closed at session shutdown, so nothing stale accumulates in your browser.
|
|
237
272
|
|
|
238
273
|
---
|
|
239
274
|
|
|
240
|
-
##
|
|
275
|
+
## Extending
|
|
241
276
|
|
|
242
|
-
When the built-in tools
|
|
277
|
+
When the built-in tools can't express a workflow, write a script and run it. It executes inside the
|
|
278
|
+
harness process with a live CDP handle — far faster than chaining tool calls.
|
|
243
279
|
|
|
244
280
|
```js
|
|
245
|
-
write("
|
|
281
|
+
write("<tmpdir>/scrape.js", `
|
|
246
282
|
const results = [];
|
|
247
283
|
for (const url of params.urls) {
|
|
248
284
|
await daemon.session().call("Page.navigate", { url });
|
|
249
285
|
await new Promise(r => setTimeout(r, 2000));
|
|
250
|
-
|
|
251
|
-
results.push({ url, title });
|
|
286
|
+
results.push({ url, title: await daemon.evaluateJs("document.title") });
|
|
252
287
|
}
|
|
253
288
|
return { content: [{ type: "text", text: JSON.stringify(results, null, 2) }] };
|
|
254
289
|
`)
|
|
255
290
|
|
|
256
|
-
browser_run_script({ path: "
|
|
291
|
+
browser_run_script({ path: "<tmpdir>/scrape.js", params: { urls: [...] } })
|
|
257
292
|
```
|
|
258
293
|
|
|
259
|
-
**Bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`,
|
|
294
|
+
**Bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`,
|
|
295
|
+
`JSON`, `Buffer`, `setTimeout`, `clearTimeout`.
|
|
260
296
|
|
|
261
|
-
`daemon` exposes
|
|
297
|
+
`daemon` exposes `evaluateJs`, `pageInfo`, `listTabs`, `switchTab`, `newTab`, `closeTab`, `current`,
|
|
298
|
+
`status`, and `session()` for raw CDP (`call` · `callOnTarget` · `callBrowser` · `takeDialog`).
|
|
262
299
|
|
|
263
|
-
Scripts
|
|
300
|
+
Scripts must live under the OS temp dir, the cwd, or `BH_SCRIPT_DIR`, and every call carries a
|
|
301
|
+
mandatory timeout (60s default, 600s max). They land on disk first, so they're reviewable and
|
|
302
|
+
re-runnable.
|
|
264
303
|
|
|
265
304
|
---
|
|
266
305
|
|
|
267
|
-
## Commands
|
|
306
|
+
## Commands, flags, and environment
|
|
307
|
+
|
|
308
|
+
| Command | Purpose |
|
|
309
|
+
|---|---|
|
|
310
|
+
| `/browser-setup` | Connect pi to your browser. Run once; idempotent. |
|
|
311
|
+
| `/browser-profile` | Pick which browser profile the agent works in. Remembered across sessions. |
|
|
312
|
+
| `/browser-status` | Connection state, selected profile, and current page. |
|
|
313
|
+
| `/deep-research <question>` | Multi-source web research → cited Markdown report. |
|
|
314
|
+
| `/browser-reload-daemon` | Force a client restart. Rarely needed — the transport reconnects on its own. |
|
|
268
315
|
|
|
269
|
-
|
|
|
270
|
-
|
|
271
|
-
|
|
|
272
|
-
|
|
|
273
|
-
|
|
|
316
|
+
| Flag / variable | Effect |
|
|
317
|
+
|---|---|
|
|
318
|
+
| `--browser-namespace <name>` | Override the session namespace (default: auto-generated). |
|
|
319
|
+
| `--browser-debug-clicks` | Save annotated click screenshots for debugging. |
|
|
320
|
+
| `BU_CDP_WS` | Attach to a remote browser by WebSocket URL instead of discovering a local one. |
|
|
321
|
+
| `BU_CDP_PORTS` | Extra ports to probe during CDP discovery, beyond 9222. |
|
|
322
|
+
| `BH_SCRIPT_DIR` | Additional directory `browser_run_script` will accept paths from. |
|
|
274
323
|
|
|
275
324
|
---
|
|
276
325
|
|
|
277
|
-
##
|
|
326
|
+
## Anti-patterns
|
|
278
327
|
|
|
279
|
-
- Don't launch your own browser
|
|
280
|
-
- Don't type credentials
|
|
281
|
-
- Don't screenshot to understand
|
|
282
|
-
|
|
283
|
-
- Don't
|
|
284
|
-
- Don't
|
|
328
|
+
- **Don't launch your own browser.** You're attached to the user's real Chrome.
|
|
329
|
+
- **Don't type credentials.** Hit an auth wall — stop and ask.
|
|
330
|
+
- **Don't screenshot to understand a page, find coordinates, or read a value.** `browser_snapshot`
|
|
331
|
+
and `browser_execute_js` are cheaper and exact.
|
|
332
|
+
- **Don't guess CSS selectors** when a snapshot ref exists.
|
|
333
|
+
- **Don't sleep to wait.** `browser_wait_for` is deterministic; `browser_wait` is a last resort.
|
|
334
|
+
- **Don't ignore dialogs.** They freeze the page's JS thread — `browser_page_info` reports them,
|
|
335
|
+
`browser_handle_dialog` clears them.
|
|
285
336
|
|
|
286
337
|
---
|
|
287
338
|
|
|
288
339
|
## Architecture
|
|
289
340
|
|
|
290
341
|
```
|
|
291
|
-
pi agent
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
342
|
+
pi agent
|
|
343
|
+
│ browser_* tools (in-process extension)
|
|
344
|
+
▼
|
|
345
|
+
pi-browser-harness ──► daemon (Unix socket / named pipe) ──► Chrome
|
|
346
|
+
CDP WebSocket
|
|
295
347
|
```
|
|
296
348
|
|
|
297
|
-
|
|
349
|
+
A long-lived daemon owns the CDP connection, so it outlives individual pi sessions — which is why
|
|
350
|
+
you only see Chrome's "Allow Remote Debugging" prompt once. `browser_run_script` executes in the
|
|
351
|
+
harness process with direct daemon and Node access.
|
|
298
352
|
|
|
299
353
|
---
|
|
300
354
|
|
|
301
355
|
## Troubleshooting
|
|
302
356
|
|
|
303
357
|
| Symptom | Fix |
|
|
304
|
-
|
|
305
|
-
| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick
|
|
306
|
-
|
|
|
307
|
-
|
|
|
308
|
-
|
|
|
309
|
-
|
|
|
310
|
-
|
|
|
358
|
+
|---|---|
|
|
359
|
+
| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick **Discover network targets**, click **Allow**. Or relaunch with `--remote-debugging-port=9222`. |
|
|
360
|
+
| `No browser instance running` | Chrome/Brave/Edge isn't open, or only helper processes remain. Open the browser, then `/browser-setup`. |
|
|
361
|
+
| `Browser harness not initialized` | Run `/browser-setup` — or let the agent call `browser_setup` itself. |
|
|
362
|
+
| `ref is stale` | The page re-rendered. Re-run `browser_snapshot` for fresh refs. |
|
|
363
|
+
| Page looks loaded but content is missing | SPA still rendering. Use `browser_wait_for({ selector })`, not a fixed sleep. |
|
|
364
|
+
| Click did nothing | Check `browser_console` for a thrown error, then `browser_network_requests` for a failed call. |
|
|
365
|
+
| Actions all hang | A JS dialog is blocking. `browser_page_info` reports it; `browser_handle_dialog` clears it. |
|
|
366
|
+
| No `[eN]` ref for your target | The element wasn't exposed as interactive. Fall back to a CSS `selector`, or `browser_execute_js` with `getBoundingClientRect()`. |
|
|
367
|
+
| Connected but the wrong browser responds | A stale `DevToolsActivePort` file. Quit every Chromium-family browser, reopen one, then `/browser-setup`. |
|
|
368
|
+
| Agent acts as the wrong account | No profile is pinned, so tabs open in whichever profile has focus. Run `/browser-profile` and pick one. |
|
|
369
|
+
| `couldn't open a window in "…" automatically` | The profile window couldn't be opened for you. Open that profile from the browser's profile menu, then retry — the harness never falls back to a different profile silently. |
|
|
311
370
|
|
|
312
371
|
---
|
|
313
372
|
|
|
314
|
-
##
|
|
373
|
+
## Security
|
|
315
374
|
|
|
316
|
-
|
|
375
|
+
This extension drives your real browser. The agent can read any page you're signed into, submit
|
|
376
|
+
forms, and act inside authenticated sessions. `browser_run_script` is full code execution in the pi
|
|
377
|
+
process with unrestricted `require` — **review any script before running it.** Scripts are
|
|
378
|
+
path-restricted and timeout-bounded, but that is a blast-radius limit, not a sandbox.
|
|
317
379
|
|
|
318
380
|
---
|
|
319
381
|
|
|
320
|
-
##
|
|
321
|
-
|
|
322
|
-
This extension drives your real Chrome. The agent can see open tabs, read page content, submit forms, and act inside authenticated sessions. `browser_run_script` evaluates JavaScript in the pi process with full `require` access — review any temporary scripts before executing them.
|
|
382
|
+
## Contributing
|
|
323
383
|
|
|
324
|
-
|
|
384
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, conventions, and the PR process.
|
|
385
|
+
Release history lives in [CHANGELOG.md](CHANGELOG.md).
|
|
325
386
|
|
|
326
387
|
## License
|
|
327
388
|
|
package/package.json
CHANGED