pi-browser-harness 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +237 -197
- package/package.json +1 -1
- package/skills/pi-browser-harness/SKILL.md +15 -0
- package/src/cdp/discovery.ts +40 -56
- package/src/cdp/session.ts +19 -0
- package/src/cdp/target-factory.ts +223 -0
- package/src/client.ts +91 -47
- package/src/domains/isolated-tab.ts +13 -14
- package/src/domains/navigate.ts +12 -19
- package/src/domains/setup.ts +4 -2
- package/src/index.ts +16 -0
- package/src/profile/bind.ts +138 -0
- package/src/profile/browser-process.ts +371 -0
- package/src/profile/command.ts +168 -0
- package/src/profile/launch.ts +90 -0
- package/src/profile/list.ts +171 -0
- package/src/profile/paths.ts +166 -0
- package/src/profile/picker.ts +73 -0
- package/src/profile/store.ts +105 -0
- package/src/setup.ts +21 -53
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,28 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to pi-browser-harness will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## 0.10.0 — 2026-07-25
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **`/browser-profile` — choose which browser profile the agent works in.** The command lists every profile in the connected browser, labelled `Name (email)` (or `Name (Profile 3)` when the profile has no signed-in account), marks the current selection, ordered the way the browser orders them. The choice is saved to `~/.pi/agent/browser-harness.json`, so it survives session termination, pi restarts, and applies across projects; a trailing `— Clear selection —` row restores the previous behavior. Picking a profile mid-session takes effect immediately — the harness closes its tabs and reopens its window in the chosen profile.
|
|
10
|
+
- **First-run profile prompt.** When no profile has been chosen, `/browser-setup` and the agent-callable `browser_setup` show the same picker before connecting, and report the result as `Browser profile: <label>`. With no interactive UI (print/RPC mode) or when the user cancels, setup continues with a one-line note and the pre-existing behavior, so non-interactive usage is unaffected.
|
|
11
|
+
- **`/browser-status` reports the selected profile.**
|
|
12
|
+
|
|
13
|
+
### Fixed
|
|
14
|
+
|
|
15
|
+
- **The agent no longer lands in an arbitrary browser profile.** Harness tabs were created with a bare `Target.createTarget`, which places them in Chrome's `defaultBrowserContextId` — a value that follows window focus. The profile the agent acted as therefore depended on which browser window the user last clicked, so the same task could run as a work account on one run and a personal account on the next. With a profile pinned, the harness opens its window inside that profile and keeps every tab there. Chrome offers no direct route for this: `Target.createTarget` rejects another profile's `browserContextId` outright, and `openerId` does not inherit the opener's context. The window is opened through the browser's own command line (`--profile-directory`, which Chromium's ProcessSingleton hands to the already-running browser) and identified by a unique `file://` sentinel page; subsequent tabs come from `window.open` evaluated with `userGesture: true`, which is the only CDP-reachable way to place a tab in a non-default browser context. When the window cannot be opened, the harness reports it and stops rather than silently using another profile.
|
|
16
|
+
- **Tab creation is funnelled through one place** (`src/cdp/target-factory.ts`). `browser_open_urls` and the isolated tabs behind `browser_web_search` / `browser_read_page` previously called `Target.createTarget` themselves, so under a pinned profile they would have run with a different profile's cookies than the visible tabs.
|
|
17
|
+
- **Snap-installed Chromium is discovered.** Ubuntu's default Chromium keeps its user data in `~/snap/chromium/common/chromium`, which was absent from the discovery list, making it invisible to the harness.
|
|
18
|
+
- **Windows profile discovery honours `%LOCALAPPDATA%`** instead of assuming `AppData\Local` under the home directory — the two diverge on roaming and managed accounts. Browsers launched with an explicit `--user-data-dir` (and Linux's `$CHROME_USER_DATA_DIR`) are now found as well.
|
|
19
|
+
- **The right browser is identified when several are running.** Browser detection now ranks candidate processes against the user-data-dir the harness is actually connected to, instead of taking the first Chromium-family process it finds; with Chrome and Brave both open, a profile window could otherwise be opened in a browser the harness is not attached to. Detection also reports the executable path and any explicit `--user-data-dir`, and on Windows uses PowerShell CIM plus the App Paths registry key rather than `wmic`, which Microsoft removed by default in Windows 11 24H2.
|
|
20
|
+
- **Installed-but-closed browsers are no longer mistaken for running ones** — liveness is derived only from live-process evidence.
|
|
21
|
+
|
|
22
|
+
### Tests
|
|
23
|
+
|
|
24
|
+
- Fixture-driven unit suites for profile enumeration, pin persistence, per-OS path resolution, and browser-process ranking (`test/profile/`), plus a real-browser end-to-end test (`test/manual/profile-e2e-test.ts`) that creates two profiles in a throwaway user-data-dir and asserts distinct browser contexts, sentinel-based window identification, and that spawned tabs stay in the pinned profile and window.
|
|
25
|
+
- CI now runs the profile suites and the end-to-end test on **ubuntu, macOS, and Windows** (Linux under Xvfb). Profile discovery is the one part of the harness whose behavior genuinely differs per OS, and the runner images ship real Chrome, so the cross-platform launch handshake is verified rather than assumed.
|
|
26
|
+
|
|
5
27
|
## 0.9.0 — 2026-07-24
|
|
6
28
|
|
|
7
29
|
### Added
|
package/README.md
CHANGED
|
@@ -2,347 +2,387 @@
|
|
|
2
2
|
|
|
3
3
|

|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
**Browser control for [pi](https://github.com/badlogic/pi-mono) agents, in the Chrome you're already using.**
|
|
6
|
+
|
|
7
|
+
Your profile, your logins, your cookies. The agent reads pages as an accessibility tree with stable
|
|
8
|
+
element handles, fills forms that React won't revert, captures network traffic and console errors,
|
|
9
|
+
searches the web, and drops to raw CDP when the tools run out. 38 tools over a single CDP connection —
|
|
10
|
+
no MCP hop, no second browser, no extra API keys.
|
|
6
11
|
|
|
7
12
|
---
|
|
8
13
|
|
|
9
|
-
##
|
|
10
|
-
|
|
11
|
-
|
|
|
12
|
-
|
|
13
|
-
| **
|
|
14
|
-
| **
|
|
15
|
-
| **
|
|
16
|
-
| **
|
|
17
|
-
| **
|
|
18
|
-
| **
|
|
19
|
-
| **
|
|
20
|
-
| **
|
|
21
|
-
| **
|
|
22
|
-
| **
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
14
|
+
## How it compares
|
|
15
|
+
|
|
16
|
+
| | **pi-browser-harness** | Playwright MCP | Puppeteer MCP | Stagehand |
|
|
17
|
+
|---|---|---|---|---|
|
|
18
|
+
| **Browser** | Your running Chrome — real profile, live sessions | Launches a clean instance | Launches a clean instance | Launches a clean instance |
|
|
19
|
+
| **Element targeting** | AX-tree refs (`[eN]`) that re-resolve at action time; coordinate and selector fallbacks | Selectors / AX locators | CSS selectors | LLM-inferred selectors |
|
|
20
|
+
| **Input dispatch** | Compositor-level CDP input — works through iframes, shadow DOM, cross-origin frames | DOM-level | DOM-level | DOM-level |
|
|
21
|
+
| **Framework-safe writes** | `browser_fill` writes via the native setter and fires bubbling events, so controlled inputs keep the value | Native fill | Manual | Native fill |
|
|
22
|
+
| **Transport** | Native pi extension, in-process | MCP / JSON-RPC round trip | MCP / JSON-RPC round trip | In-process + its own LLM calls |
|
|
23
|
+
| **Extra credentials** | None | None | None | Its own model API key |
|
|
24
|
+
| **Concurrency** | Reads run in parallel; mutations auto-serialize through a shared mutex | Sequential | Sequential | Sequential |
|
|
25
|
+
| **Isolation** | Dedicated Chrome window per session; refuses tabs it didn't open | N/A (own browser) | N/A | N/A |
|
|
26
|
+
| **Terminal output** | Inline screenshots and `Ctrl+O`-expandable tool results | Plain text | Plain text | Plain text |
|
|
27
|
+
| **Escape hatch** | `browser_run_script` — raw CDP + full Node in one call | — | — | — |
|
|
28
|
+
|
|
29
|
+
<sub>Reflects each project's default configuration. Corrections welcome via PR.</sub>
|
|
30
|
+
|
|
31
|
+
**The short version:** if you live in pi and want an agent driving the same Chrome you're already
|
|
32
|
+
signed into, this is the one that fits.
|
|
28
33
|
|
|
29
34
|
---
|
|
30
35
|
|
|
31
|
-
##
|
|
36
|
+
## Install
|
|
32
37
|
|
|
33
38
|
```bash
|
|
34
|
-
# 1. Install
|
|
35
39
|
pi install npm:pi-browser-harness
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Then enable remote debugging in your browser — either open `chrome://inspect/#remote-debugging`
|
|
43
|
+
(or `brave://inspect`, `edge://inspect`), tick **Discover network targets**, and click **Allow** —
|
|
44
|
+
or relaunch the browser with `--remote-debugging-port=9222`.
|
|
45
|
+
|
|
46
|
+
Finally, run `/browser-setup` in pi to connect. It's a one-time step — the connection survives across
|
|
47
|
+
pi sessions and reconnects on its own when Chrome restarts.
|
|
36
48
|
|
|
37
|
-
|
|
38
|
-
# Open chrome://inspect/#remote-debugging in Chrome,
|
|
39
|
-
# tick "Discover network targets", click Allow.
|
|
40
|
-
# Or launch Chrome with --remote-debugging-port=9222
|
|
49
|
+
The first time you connect, you'll be asked which browser profile the agent should work in:
|
|
41
50
|
|
|
42
|
-
|
|
43
|
-
|
|
51
|
+
```
|
|
52
|
+
Select the browser profile the agent should use
|
|
53
|
+
vertexcover.io (aman@vertexcover.io)
|
|
54
|
+
✓ Aman (personal@gmail.com)
|
|
55
|
+
Harness (harness@gmail.com)
|
|
56
|
+
— Clear selection (use whichever window is focused) —
|
|
44
57
|
```
|
|
45
58
|
|
|
46
|
-
|
|
59
|
+
That choice is saved to `~/.pi/agent/browser-harness.json` and reused by every project until you
|
|
60
|
+
change it with `/browser-profile`. It decides which logins, cookies, and extensions the agent works
|
|
61
|
+
with, so pinning it keeps runs reproducible. Skip the prompt and the harness behaves as it always
|
|
62
|
+
did: tabs open in whichever profile currently has focus.
|
|
47
63
|
|
|
48
|
-
|
|
49
|
-
- Node.js ≥ 22
|
|
50
|
-
- Chrome / Chromium / Edge
|
|
64
|
+
**Requires:** pi (latest) · Node.js ≥ 22 · Chrome, Chromium, Brave, or Edge
|
|
51
65
|
|
|
52
66
|
---
|
|
53
67
|
|
|
54
|
-
##
|
|
68
|
+
## The mental model
|
|
69
|
+
|
|
70
|
+
Interaction is **ref-first**. `browser_snapshot` returns the accessibility tree, and every
|
|
71
|
+
interactive element carries a stable ref (`[eN]`) plus click coordinates `@(x,y)`:
|
|
55
72
|
|
|
56
73
|
```
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
│ → browser_snapshot (DEFAULT — AX tree with @(x,y) per interactive element)
|
|
61
|
-
│
|
|
62
|
-
├─ A specific element's value / attribute / coords?
|
|
63
|
-
│ → browser_execute_js (e.g. el.innerText, el.getBoundingClientRect())
|
|
64
|
-
│
|
|
65
|
-
├─ Network behavior on the current page?
|
|
66
|
-
│ → browser_network_requests
|
|
67
|
-
│
|
|
68
|
-
└─ Visual rendering (layout / colors / chart drew correctly)?
|
|
69
|
-
→ browser_screenshot (LAST RESORT — pixels only)
|
|
74
|
+
[e7] textbox "Email" @(412,288)
|
|
75
|
+
[e9] combobox "Country" @(412,344)
|
|
76
|
+
[e12] button "Create account" @(456,412)
|
|
70
77
|
```
|
|
71
78
|
|
|
72
|
-
Pass
|
|
79
|
+
Pass the ref — not the coordinates — to `browser_click`, `browser_fill`, `browser_select_option`,
|
|
80
|
+
`browser_focus`, `browser_upload_file`, or `browser_dispatch_key`. Refs are keyed to element
|
|
81
|
+
identity, so they re-resolve position at action time and survive the re-renders that SPA forms
|
|
82
|
+
trigger on every keystroke. Coordinates go stale the moment the layout reflows.
|
|
83
|
+
|
|
84
|
+
Three things follow from that:
|
|
85
|
+
|
|
86
|
+
- **Never screenshot to find a click target.** The snapshot already has refs and coordinates.
|
|
87
|
+
- **Mutating calls append a "Page changes" diff.** Read it to confirm the step landed — a panel
|
|
88
|
+
closed, a new `[eN]` appeared — instead of taking a screenshot to check.
|
|
89
|
+
- **`"ref is stale"` means the page changed.** Re-snapshot for fresh refs; don't retry blindly.
|
|
90
|
+
|
|
91
|
+
### Which tool answers your question
|
|
92
|
+
|
|
93
|
+
| You need to know… | Reach for |
|
|
94
|
+
|---|---|
|
|
95
|
+
| What's on the page, what's clickable | `browser_snapshot` — the default |
|
|
96
|
+
| One element's text, value, or geometry | `browser_execute_js` — one round trip |
|
|
97
|
+
| An article's readable content | `browser_read_page` |
|
|
98
|
+
| Why nothing happened after an action | `browser_console` |
|
|
99
|
+
| What the page requested over the network | `browser_network_requests` |
|
|
100
|
+
| Whether something *rendered* correctly | `browser_screenshot` — last resort, pixels only |
|
|
73
101
|
|
|
74
102
|
---
|
|
75
103
|
|
|
76
104
|
## Tools
|
|
77
105
|
|
|
78
|
-
###
|
|
106
|
+
### Reading the page
|
|
79
107
|
|
|
80
108
|
| Tool | Purpose |
|
|
81
|
-
|
|
82
|
-
| `browser_snapshot` | **Default
|
|
83
|
-
| `browser_execute_js` | Surgical DOM reads —
|
|
84
|
-
| `
|
|
85
|
-
| `
|
|
86
|
-
| `
|
|
109
|
+
|---|---|
|
|
110
|
+
| `browser_snapshot` | **Default.** Accessibility tree with `[eN]` refs and `@(x,y)` per interactive element. Optional `includeScreenshot`. |
|
|
111
|
+
| `browser_execute_js` | Surgical DOM reads — text, attributes, `getBoundingClientRect()`. Cheapest precise read. |
|
|
112
|
+
| `browser_page_info` | URL, title, viewport, scroll position — or the pending JS dialog, if one is blocking. |
|
|
113
|
+
| `browser_console` | JS errors and console output. Pass `sinceSeq` from the previous `nextCursor` to see only what an action caused. |
|
|
114
|
+
| `browser_screenshot` | PNG/JPEG capture. For verifying visual rendering only. |
|
|
87
115
|
|
|
88
|
-
###
|
|
116
|
+
### Web research
|
|
89
117
|
|
|
90
118
|
| Tool | Purpose |
|
|
91
|
-
|
|
92
|
-
| `browser_web_search` |
|
|
93
|
-
| `browser_read_page` | Reader mode
|
|
119
|
+
|---|---|
|
|
120
|
+
| `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real Google SERP in your Chrome. No API key. Links only. |
|
|
121
|
+
| `browser_read_page` | Reader mode. A `url` (opened and closed in its own tab) or an owned `targetId` → clean article text, boilerplate stripped. |
|
|
94
122
|
|
|
95
|
-
|
|
123
|
+
Both run in isolated tabs and never disturb your current page.
|
|
124
|
+
|
|
125
|
+
### Interacting
|
|
126
|
+
|
|
127
|
+
| Tool | Purpose |
|
|
128
|
+
|---|---|
|
|
129
|
+
| `browser_click` | Click by `ref` (preferred) or viewport `x`/`y`. Compositor-level, so iframes and shadow DOM work. |
|
|
130
|
+
| `browser_fill` | Set an input, textarea, or contenteditable. Fires bubbling `input`/`change`, returns the value written. |
|
|
131
|
+
| `browser_select_option` | Choose in a native `<select>` by value, label, or index. Returns available options on no match. |
|
|
132
|
+
| `browser_focus` | Deterministic `.focus()` — no coordinate accuracy needed. |
|
|
133
|
+
| `browser_type` | Type into the focused element. For keystroke-sensitive widgets; prefer `browser_fill` otherwise. |
|
|
134
|
+
| `browser_press_key` | Press a key with an optional modifier bitfield. |
|
|
135
|
+
| `browser_dispatch_key` | Synthetic DOM `KeyboardEvent` on one element, for pages that ignore raw CDP keys. |
|
|
136
|
+
| `browser_scroll` | Scroll by delta. W3C wheel convention: positive `deltaY` = down. |
|
|
137
|
+
| `browser_drag_and_drop` | Drag between two coordinate pairs. |
|
|
138
|
+
|
|
139
|
+
### Navigation and tabs
|
|
96
140
|
|
|
97
141
|
| Tool | Purpose |
|
|
98
|
-
|
|
99
|
-
| `
|
|
142
|
+
|---|---|
|
|
143
|
+
| `browser_navigate` | Go to a URL in the current tab. |
|
|
144
|
+
| `browser_new_tab` / `browser_open_urls` | Open one tab, or many URLs in parallel. |
|
|
145
|
+
| `browser_go_back` / `browser_go_forward` / `browser_reload` | History. |
|
|
146
|
+
| `browser_list_tabs` / `browser_current_tab` | List tabs (`scope:"owned"` by default) and inspect the attached one. |
|
|
147
|
+
| `browser_switch_tab` / `browser_close_tab` | Switch or close — owned tabs only. |
|
|
100
148
|
|
|
101
|
-
###
|
|
149
|
+
### Waiting and dialogs
|
|
102
150
|
|
|
103
151
|
| Tool | Purpose |
|
|
104
|
-
|
|
105
|
-
| `
|
|
106
|
-
| `
|
|
107
|
-
| `
|
|
108
|
-
| `
|
|
152
|
+
|---|---|
|
|
153
|
+
| `browser_wait_for` | Wait until a selector appears (or disappears with `gone:true`), or text shows up. **Prefer this over sleeping.** |
|
|
154
|
+
| `browser_wait_for_load` | Wait for `document.readyState === "complete"`. |
|
|
155
|
+
| `browser_wait` | Fixed delay, in seconds. |
|
|
156
|
+
| `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt`. |
|
|
109
157
|
|
|
110
|
-
###
|
|
158
|
+
### Network
|
|
111
159
|
|
|
112
160
|
| Tool | Purpose |
|
|
113
|
-
|
|
114
|
-
| `
|
|
115
|
-
| `
|
|
116
|
-
| `browser_press_key` | Press a key with optional modifiers |
|
|
117
|
-
| `browser_scroll` | Scroll the page by delta pixels |
|
|
161
|
+
|---|---|
|
|
162
|
+
| `browser_network_requests` | Requests captured on the current tab since attach. Filter by URL, method, status, resource type, or recency; optional response bodies. |
|
|
163
|
+
| `browser_http_get` | Plain GET outside the browser. Much faster than navigate-then-read for JSON APIs — but no JS rendering. |
|
|
118
164
|
|
|
119
|
-
###
|
|
165
|
+
### Files and viewport
|
|
120
166
|
|
|
121
167
|
| Tool | Purpose |
|
|
122
|
-
|
|
123
|
-
| `
|
|
124
|
-
| `
|
|
125
|
-
| `
|
|
168
|
+
|---|---|
|
|
169
|
+
| `browser_upload_file` | Set files on a file input, with a DataTransfer fallback for stubborn pages. |
|
|
170
|
+
| `browser_download` | Set the download directory and suppress the save-as prompt. |
|
|
171
|
+
| `browser_print_to_pdf` | Print the page to PDF via `Page.printToPDF`. |
|
|
172
|
+
| `browser_viewport_resize` | Override viewport size and device pixel ratio for responsive checks. |
|
|
126
173
|
|
|
127
|
-
|
|
174
|
+
### Escape hatches
|
|
128
175
|
|
|
129
|
-
|
|
176
|
+
| Tool | Purpose |
|
|
177
|
+
|---|---|
|
|
178
|
+
| `browser_run_script` | Run a temporary script with raw CDP and full Node access. See [Extending](#extending). |
|
|
179
|
+
| `browser_setup` | Agent-callable setup. Idempotent — how the agent recovers from `not initialized` on its own. |
|
|
130
180
|
|
|
131
|
-
|
|
181
|
+
`browser_snapshot`, `browser_network_requests`, `browser_console`, `browser_execute_js`,
|
|
182
|
+
`browser_web_search`, and `browser_read_page` ship custom TUI rendering: compact by default,
|
|
183
|
+
full output on **`Ctrl+O`** (`app.tools.expand`). The model always receives the complete result
|
|
184
|
+
regardless of render state.
|
|
132
185
|
|
|
133
|
-
|
|
186
|
+
---
|
|
134
187
|
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
```
|
|
188
|
+
## Recipes
|
|
189
|
+
|
|
190
|
+
### Fill a form on an SPA
|
|
139
191
|
|
|
140
|
-
|
|
192
|
+
Wait, snapshot once, then act by ref. No screenshots anywhere in this loop.
|
|
141
193
|
|
|
142
194
|
```
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
browser_snapshot() # verify next state
|
|
195
|
+
browser_wait_for({ selector: "#email" })
|
|
196
|
+
browser_snapshot() // → [e7] textbox, [e9] combobox, [e12] button
|
|
197
|
+
browser_fill({ ref: "e7", value: "a@b.com" })
|
|
198
|
+
browser_select_option({ ref: "e9", label: "India" })
|
|
199
|
+
browser_click({ ref: "e12" }) // re-resolves position even after reflow
|
|
149
200
|
```
|
|
150
201
|
|
|
151
|
-
|
|
202
|
+
Each mutating call returns a "Page changes" diff — that's your confirmation the step landed.
|
|
203
|
+
|
|
204
|
+
### Extract data
|
|
152
205
|
|
|
153
206
|
```js
|
|
154
207
|
// One value
|
|
155
208
|
browser_execute_js({ expression: "document.querySelector('.price').innerText" })
|
|
156
209
|
|
|
157
|
-
//
|
|
158
|
-
browser_http_get({ url: "https://api.github.com/repos/amankumarsingh77/pi-browser-harness" })
|
|
159
|
-
|
|
160
|
-
// Structured arrays
|
|
210
|
+
// Structured
|
|
161
211
|
browser_execute_js({ expression: `JSON.stringify(
|
|
162
212
|
Array.from(document.querySelectorAll('.result')).map(el => ({
|
|
163
213
|
title: el.querySelector('h3')?.textContent,
|
|
164
214
|
link: el.querySelector('a')?.href,
|
|
165
215
|
}))
|
|
166
216
|
)` })
|
|
217
|
+
|
|
218
|
+
// Skip the browser entirely for a JSON endpoint
|
|
219
|
+
browser_http_get({ url: "https://api.github.com/repos/amankumarsingh77/pi-browser-harness" })
|
|
167
220
|
```
|
|
168
221
|
|
|
169
|
-
###
|
|
222
|
+
### Debug a failing page
|
|
170
223
|
|
|
171
224
|
```
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
browser_network_requests({
|
|
225
|
+
browser_click({ ref: "e12" })
|
|
226
|
+
browser_console({ levels: ["error", "warn"] }) // did JS throw?
|
|
227
|
+
browser_network_requests({ // did a request 4xx/5xx?
|
|
175
228
|
urlPattern: "/api/",
|
|
176
229
|
statusFilter: { min: 400 },
|
|
177
230
|
includeResponseBodies: true
|
|
178
231
|
})
|
|
179
232
|
```
|
|
180
233
|
|
|
181
|
-
###
|
|
182
|
-
|
|
183
|
-
Search → read, with each tool running in its own isolated tab (your current tab is never disturbed):
|
|
234
|
+
### Search, then read
|
|
184
235
|
|
|
185
236
|
```
|
|
186
237
|
browser_web_search({ query: "chrome devtools protocol overview", limit: 5 })
|
|
187
|
-
# → ranked {title, url, snippet, rank} — links only
|
|
188
238
|
browser_read_page({ url: "<the best result's url>" })
|
|
189
|
-
# → clean main-article text, boilerplate stripped
|
|
190
|
-
```
|
|
191
|
-
|
|
192
|
-
### Deep research
|
|
193
|
-
|
|
194
|
-
For a question that needs multiple sources synthesized into a cited report, run:
|
|
195
|
-
|
|
196
|
-
```
|
|
197
|
-
/deep-research <your question>
|
|
198
239
|
```
|
|
199
240
|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
`
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
### Visual verification (only when pixels matter)
|
|
207
|
-
|
|
208
|
-
```
|
|
209
|
-
browser_click({ x, y }) # got coords from browser_snapshot
|
|
210
|
-
browser_snapshot() # confirm the form transitioned
|
|
211
|
-
browser_screenshot() # ONLY if you need to verify a chart/modal/CSS rendered correctly
|
|
212
|
-
```
|
|
241
|
+
For a question that needs several sources synthesized into a **cited report**, use
|
|
242
|
+
`/deep-research <question>`. It decomposes the question, fans out one isolated
|
|
243
|
+
`web-search-researcher` subagent per sub-question, loops for coverage under a hard ceiling, and
|
|
244
|
+
writes a Markdown report where every claim carries a source link. The skill also auto-triggers on
|
|
245
|
+
research-shaped requests without the command.
|
|
213
246
|
|
|
214
247
|
### Keyboard modifiers
|
|
215
248
|
|
|
216
|
-
|
|
217
|
-
|-----|-----|
|
|
218
|
-
| Alt | 1 |
|
|
219
|
-
| Ctrl | 2 |
|
|
220
|
-
| Meta / Cmd | 4 |
|
|
221
|
-
| Shift | 8 |
|
|
249
|
+
Modifiers are a bitfield: `1` Alt · `2` Ctrl · `4` Meta/Cmd · `8` Shift.
|
|
222
250
|
|
|
223
251
|
```
|
|
224
252
|
browser_press_key({ key: "c", modifiers: 2 }) // Ctrl+C
|
|
225
|
-
browser_press_key({ key: "v", modifiers: 4 }) // Cmd+V
|
|
226
253
|
browser_press_key({ key: "T", modifiers: 10 }) // Ctrl+Shift+T
|
|
227
254
|
```
|
|
228
255
|
|
|
229
|
-
### Dialogs
|
|
230
|
-
|
|
231
|
-
JS dialogs freeze the page. Check `browser_page_info` first — if it reports a dialog, handle it before anything else:
|
|
232
|
-
|
|
233
|
-
```
|
|
234
|
-
browser_handle_dialog({ action: "accept" }) // confirm
|
|
235
|
-
browser_handle_dialog({ action: "dismiss" }) // cancel
|
|
236
|
-
browser_handle_dialog({ action: "accept", promptText: "hello" }) // prompt
|
|
237
|
-
```
|
|
238
|
-
|
|
239
256
|
---
|
|
240
257
|
|
|
241
|
-
##
|
|
258
|
+
## Concurrency and isolation
|
|
242
259
|
|
|
243
|
-
|
|
260
|
+
**Parallel reads, serialized writes.** Observation tools run concurrently with each other and with
|
|
261
|
+
mutations. Mutations (`click`, `fill`, `type`, `scroll`, `navigate`, `switch_tab`, …) pass through a
|
|
262
|
+
shared async mutex and FIFO-queue automatically, so emitting several in one turn is safe.
|
|
244
263
|
|
|
245
264
|
```
|
|
246
|
-
|
|
247
|
-
browser_snapshot()
|
|
248
|
-
browser_network_requests({ sinceMs: 5000 })
|
|
249
|
-
browser_http_get({ url: "..." })
|
|
265
|
+
browser_snapshot() + browser_network_requests({ sinceMs: 5000 }) + browser_http_get({ url: "..." })
|
|
250
266
|
```
|
|
251
267
|
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
The harness never touches tabs you didn't open through it. On first attach it spawns a dedicated Chrome window; subsequent `browser_new_tab` calls open inside that window. `browser_list_tabs` defaults to `scope:"owned"` (pass `scope:"all"` to see read-only listings of your other tabs); `browser_switch_tab` and `browser_close_tab` refuse non-owned tabs.
|
|
268
|
+
**Tab ownership.** On first attach the harness opens a dedicated Chrome window and only operates
|
|
269
|
+
inside it. `browser_list_tabs` defaults to `scope:"owned"` (pass `scope:"all"` for a read-only view
|
|
270
|
+
of your other tabs); `browser_switch_tab` and `browser_close_tab` refuse anything this session
|
|
271
|
+
didn't open. Owned tabs are closed at session shutdown, so nothing stale accumulates in your browser.
|
|
257
272
|
|
|
258
273
|
---
|
|
259
274
|
|
|
260
|
-
##
|
|
275
|
+
## Extending
|
|
261
276
|
|
|
262
|
-
When the built-in tools
|
|
277
|
+
When the built-in tools can't express a workflow, write a script and run it. It executes inside the
|
|
278
|
+
harness process with a live CDP handle — far faster than chaining tool calls.
|
|
263
279
|
|
|
264
280
|
```js
|
|
265
|
-
write("
|
|
281
|
+
write("<tmpdir>/scrape.js", `
|
|
266
282
|
const results = [];
|
|
267
283
|
for (const url of params.urls) {
|
|
268
284
|
await daemon.session().call("Page.navigate", { url });
|
|
269
285
|
await new Promise(r => setTimeout(r, 2000));
|
|
270
|
-
|
|
271
|
-
results.push({ url, title });
|
|
286
|
+
results.push({ url, title: await daemon.evaluateJs("document.title") });
|
|
272
287
|
}
|
|
273
288
|
return { content: [{ type: "text", text: JSON.stringify(results, null, 2) }] };
|
|
274
289
|
`)
|
|
275
290
|
|
|
276
|
-
browser_run_script({ path: "
|
|
291
|
+
browser_run_script({ path: "<tmpdir>/scrape.js", params: { urls: [...] } })
|
|
277
292
|
```
|
|
278
293
|
|
|
279
|
-
**Bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`,
|
|
294
|
+
**Bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`,
|
|
295
|
+
`JSON`, `Buffer`, `setTimeout`, `clearTimeout`.
|
|
280
296
|
|
|
281
|
-
`daemon` exposes
|
|
297
|
+
`daemon` exposes `evaluateJs`, `pageInfo`, `listTabs`, `switchTab`, `newTab`, `closeTab`, `current`,
|
|
298
|
+
`status`, and `session()` for raw CDP (`call` · `callOnTarget` · `callBrowser` · `takeDialog`).
|
|
282
299
|
|
|
283
|
-
Scripts
|
|
300
|
+
Scripts must live under the OS temp dir, the cwd, or `BH_SCRIPT_DIR`, and every call carries a
|
|
301
|
+
mandatory timeout (60s default, 600s max). They land on disk first, so they're reviewable and
|
|
302
|
+
re-runnable.
|
|
284
303
|
|
|
285
304
|
---
|
|
286
305
|
|
|
287
|
-
## Commands
|
|
306
|
+
## Commands, flags, and environment
|
|
307
|
+
|
|
308
|
+
| Command | Purpose |
|
|
309
|
+
|---|---|
|
|
310
|
+
| `/browser-setup` | Connect pi to your browser. Run once; idempotent. |
|
|
311
|
+
| `/browser-profile` | Pick which browser profile the agent works in. Remembered across sessions. |
|
|
312
|
+
| `/browser-status` | Connection state, selected profile, and current page. |
|
|
313
|
+
| `/deep-research <question>` | Multi-source web research → cited Markdown report. |
|
|
314
|
+
| `/browser-reload-daemon` | Force a client restart. Rarely needed — the transport reconnects on its own. |
|
|
288
315
|
|
|
289
|
-
|
|
|
290
|
-
|
|
291
|
-
|
|
|
292
|
-
|
|
|
293
|
-
|
|
|
294
|
-
|
|
|
316
|
+
| Flag / variable | Effect |
|
|
317
|
+
|---|---|
|
|
318
|
+
| `--browser-namespace <name>` | Override the session namespace (default: auto-generated). |
|
|
319
|
+
| `--browser-debug-clicks` | Save annotated click screenshots for debugging. |
|
|
320
|
+
| `BU_CDP_WS` | Attach to a remote browser by WebSocket URL instead of discovering a local one. |
|
|
321
|
+
| `BU_CDP_PORTS` | Extra ports to probe during CDP discovery, beyond 9222. |
|
|
322
|
+
| `BH_SCRIPT_DIR` | Additional directory `browser_run_script` will accept paths from. |
|
|
295
323
|
|
|
296
324
|
---
|
|
297
325
|
|
|
298
|
-
##
|
|
326
|
+
## Anti-patterns
|
|
299
327
|
|
|
300
|
-
- Don't launch your own browser
|
|
301
|
-
- Don't type credentials
|
|
302
|
-
- Don't screenshot to understand
|
|
303
|
-
|
|
304
|
-
- Don't
|
|
305
|
-
- Don't
|
|
328
|
+
- **Don't launch your own browser.** You're attached to the user's real Chrome.
|
|
329
|
+
- **Don't type credentials.** Hit an auth wall — stop and ask.
|
|
330
|
+
- **Don't screenshot to understand a page, find coordinates, or read a value.** `browser_snapshot`
|
|
331
|
+
and `browser_execute_js` are cheaper and exact.
|
|
332
|
+
- **Don't guess CSS selectors** when a snapshot ref exists.
|
|
333
|
+
- **Don't sleep to wait.** `browser_wait_for` is deterministic; `browser_wait` is a last resort.
|
|
334
|
+
- **Don't ignore dialogs.** They freeze the page's JS thread — `browser_page_info` reports them,
|
|
335
|
+
`browser_handle_dialog` clears them.
|
|
306
336
|
|
|
307
337
|
---
|
|
308
338
|
|
|
309
339
|
## Architecture
|
|
310
340
|
|
|
311
341
|
```
|
|
312
|
-
pi agent
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
342
|
+
pi agent
|
|
343
|
+
│ browser_* tools (in-process extension)
|
|
344
|
+
▼
|
|
345
|
+
pi-browser-harness ──► daemon (Unix socket / named pipe) ──► Chrome
|
|
346
|
+
CDP WebSocket
|
|
316
347
|
```
|
|
317
348
|
|
|
318
|
-
|
|
349
|
+
A long-lived daemon owns the CDP connection, so it outlives individual pi sessions — which is why
|
|
350
|
+
you only see Chrome's "Allow Remote Debugging" prompt once. `browser_run_script` executes in the
|
|
351
|
+
harness process with direct daemon and Node access.
|
|
319
352
|
|
|
320
353
|
---
|
|
321
354
|
|
|
322
355
|
## Troubleshooting
|
|
323
356
|
|
|
324
357
|
| Symptom | Fix |
|
|
325
|
-
|
|
326
|
-
| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick
|
|
327
|
-
|
|
|
328
|
-
|
|
|
329
|
-
|
|
|
330
|
-
|
|
|
331
|
-
|
|
|
358
|
+
|---|---|
|
|
359
|
+
| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick **Discover network targets**, click **Allow**. Or relaunch with `--remote-debugging-port=9222`. |
|
|
360
|
+
| `No browser instance running` | Chrome/Brave/Edge isn't open, or only helper processes remain. Open the browser, then `/browser-setup`. |
|
|
361
|
+
| `Browser harness not initialized` | Run `/browser-setup` — or let the agent call `browser_setup` itself. |
|
|
362
|
+
| `ref is stale` | The page re-rendered. Re-run `browser_snapshot` for fresh refs. |
|
|
363
|
+
| Page looks loaded but content is missing | SPA still rendering. Use `browser_wait_for({ selector })`, not a fixed sleep. |
|
|
364
|
+
| Click did nothing | Check `browser_console` for a thrown error, then `browser_network_requests` for a failed call. |
|
|
365
|
+
| Actions all hang | A JS dialog is blocking. `browser_page_info` reports it; `browser_handle_dialog` clears it. |
|
|
366
|
+
| No `[eN]` ref for your target | The element wasn't exposed as interactive. Fall back to a CSS `selector`, or `browser_execute_js` with `getBoundingClientRect()`. |
|
|
367
|
+
| Connected but the wrong browser responds | A stale `DevToolsActivePort` file. Quit every Chromium-family browser, reopen one, then `/browser-setup`. |
|
|
368
|
+
| Agent acts as the wrong account | No profile is pinned, so tabs open in whichever profile has focus. Run `/browser-profile` and pick one. |
|
|
369
|
+
| `couldn't open a window in "…" automatically` | The profile window couldn't be opened for you. Open that profile from the browser's profile menu, then retry — the harness never falls back to a different profile silently. |
|
|
332
370
|
|
|
333
371
|
---
|
|
334
372
|
|
|
335
|
-
##
|
|
373
|
+
## Security
|
|
336
374
|
|
|
337
|
-
|
|
375
|
+
This extension drives your real browser. The agent can read any page you're signed into, submit
|
|
376
|
+
forms, and act inside authenticated sessions. `browser_run_script` is full code execution in the pi
|
|
377
|
+
process with unrestricted `require` — **review any script before running it.** Scripts are
|
|
378
|
+
path-restricted and timeout-bounded, but that is a blast-radius limit, not a sandbox.
|
|
338
379
|
|
|
339
380
|
---
|
|
340
381
|
|
|
341
|
-
##
|
|
342
|
-
|
|
343
|
-
This extension drives your real Chrome. The agent can see open tabs, read page content, submit forms, and act inside authenticated sessions. `browser_run_script` evaluates JavaScript in the pi process with full `require` access — review any temporary scripts before executing them.
|
|
382
|
+
## Contributing
|
|
344
383
|
|
|
345
|
-
|
|
384
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, conventions, and the PR process.
|
|
385
|
+
Release history lives in [CHANGELOG.md](CHANGELOG.md).
|
|
346
386
|
|
|
347
387
|
## License
|
|
348
388
|
|