pi-browser-harness 0.12.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +56 -0
- package/README.md +132 -0
- package/package.json +15 -7
- package/skills/pi-browser-harness/SKILL.md +89 -2
- package/src/cdp/commands.ts +205 -14
- package/src/cdp/console-buffer.ts +37 -23
- package/src/cdp/discovery.ts +57 -25
- package/src/cdp/events.ts +9 -3
- package/src/cdp/network-buffer.ts +38 -32
- package/src/cdp/ownership.ts +22 -4
- package/src/cdp/record-store.ts +2 -2
- package/src/cdp/session.ts +358 -216
- package/src/cdp/target-factory.ts +34 -15
- package/src/cdp/types.ts +17 -1
- package/src/client.ts +127 -54
- package/src/daemon/bridge.ts +110 -349
- package/src/daemon/connection.ts +323 -0
- package/src/daemon/index.ts +77 -48
- package/src/daemon/lines.ts +40 -0
- package/src/daemon/main.ts +12 -0
- package/src/daemon/protocol.ts +55 -25
- package/src/daemon/server.ts +100 -63
- package/src/daemon/socket.ts +10 -0
- package/src/daemon/spawn.ts +21 -22
- package/src/daemon/transport.ts +80 -49
- package/src/domains/ax-tree.ts +27 -8
- package/src/domains/cdp-call.ts +14 -7
- package/src/domains/click.ts +2 -2
- package/src/domains/concurrency.ts +50 -0
- package/src/domains/console.ts +20 -64
- package/src/domains/drag.ts +26 -28
- package/src/domains/element-call.ts +102 -68
- package/src/domains/files.ts +13 -9
- package/src/domains/form-batch.ts +13 -10
- package/src/domains/form.ts +78 -107
- package/src/domains/headers.ts +2 -2
- package/src/domains/history.ts +4 -4
- package/src/domains/io.ts +35 -0
- package/src/domains/isolated-tab.ts +6 -12
- package/src/domains/js.ts +67 -46
- package/src/domains/keyboard.ts +14 -11
- package/src/domains/navigate.ts +102 -66
- package/src/domains/network.ts +55 -95
- package/src/domains/page/fill.page.ts +139 -0
- package/src/domains/page/form.page.ts +61 -0
- package/src/domains/page/inject.ts +37 -0
- package/src/domains/page/keyboard.page.ts +16 -0
- package/src/domains/page/live-value.page.ts +10 -0
- package/src/domains/page/read-page.page.ts +51 -0
- package/src/domains/page/recipe.page.ts +82 -0
- package/src/domains/page/serp.page.ts +42 -0
- package/src/domains/page.ts +18 -51
- package/src/domains/poll.ts +42 -0
- package/src/domains/readpage/read-page.ts +48 -53
- package/src/domains/ref-resolve.ts +43 -29
- package/src/domains/ref-signature.ts +11 -0
- package/src/domains/render.ts +12 -23
- package/src/domains/screenshot-capture.ts +4 -2
- package/src/domains/screenshot.ts +13 -10
- package/src/domains/scroll.ts +4 -4
- package/src/domains/search/google-serp.ts +78 -8
- package/src/domains/search/redirect-resolver.ts +41 -0
- package/src/domains/search/web-search.ts +103 -40
- package/src/domains/snapshot.ts +81 -78
- package/src/domains/tab-resolve.ts +36 -0
- package/src/domains/tabs.ts +18 -69
- package/src/domains/target.ts +1 -1
- package/src/index.ts +101 -33
- package/src/profile/bind.ts +45 -14
- package/src/profile/browser-process.ts +50 -12
- package/src/profile/command.ts +20 -12
- package/src/profile/launch.ts +13 -9
- package/src/profile/list.ts +9 -13
- package/src/profile/store.ts +2 -1
- package/src/prompt.ts +34 -8
- package/src/recipes/command.ts +96 -0
- package/src/recipes/compile.ts +198 -0
- package/src/recipes/find.ts +49 -0
- package/src/recipes/journal.ts +125 -0
- package/src/recipes/recipe.ts +215 -0
- package/src/recipes/recordable.ts +359 -0
- package/src/recipes/replay.ts +354 -0
- package/src/recipes/run-tool.ts +132 -0
- package/src/recipes/save-tool.ts +247 -0
- package/src/recipes/store.ts +103 -0
- package/src/recipes/trust.ts +172 -0
- package/src/registry.ts +69 -4
- package/src/schemas/parse.ts +2 -2
- package/src/setup.ts +28 -19
- package/src/state.ts +2 -1
- package/src/util/debug.ts +7 -1
- package/src/util/guards.ts +3 -0
- package/src/util/js-source.ts +28 -0
- package/src/util/mutex.ts +1 -1
- package/src/util/paths.ts +3 -2
- package/src/util/result.ts +8 -0
- package/src/util/sharp-shim.ts +18 -11
- package/src/util/sleep.ts +18 -0
- package/src/util/tool.ts +185 -114
- package/src/util/truncate.ts +1 -1
- package/tsconfig.json +2 -1
- package/src/domains/fill-engine.ts +0 -80
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,62 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to pi-browser-harness will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## 0.14.0 — 2026-11-01
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **Browser recipes: journal and save.** Every page-changing browser call is journaled per session, and `browser_save_recipe` saves the agent's last browser task as a YAML recipe with `{{params}}`, after the user confirms it.
|
|
10
|
+
- **Browser recipes: replay.** `browser_run_recipe` and `/browser-recipe NAME key=value` replay a saved recipe with no model calls, finding each element by role and name and checking each step's recorded outcome. Downloads and status/alert toasts are now recorded, so a step can expect them.
|
|
11
|
+
- **Browser recipes: finish and repair.** A failed replay hands the agent the failed step and a snapshot; the agent finishes the task, then `browser_save_recipe` with `repair_step` rewrites the recipe from that step, keeping the previous version. An auth wall stops the replay without typing anything.
|
|
12
|
+
- **Browser recipes: project recipes need approval.** A recipe from the project's `.pi/browser-recipes/` asks for confirmation, listing its URLs, uploads and downloads, before its first run and again whenever its file changes.
|
|
13
|
+
- **Browser recipes: read steps return data.** `browser_http_get` and `browser_read_page` of a URL are kept as recipe steps, and a replay returns what they fetched: in `browser_run_recipe`'s result, and as a session message from `/browser-recipe` that starts no model turn. A read made with request headers or a tab id is left out.
|
|
14
|
+
- **Browser recipes: bot checks.** A replay waits up to 30s while Cloudflare's "Just a moment..." check is showing; if it never clears, the replay stops and the agent asks the user to complete it.
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
|
|
18
|
+
- **`browser_execute_js` with a top-level `return` returns its value.** Any body with a top-level `return` runs inside an async function that is awaited, and scripts run in REPL mode, so a later call may declare the same `const` or `let` again.
|
|
19
|
+
- **`browser_close_tab` is fast and confirms the close.** Closing the current tab no longer waits 10s, and a tab Chrome keeps open (one closed while still loading) is closed again, or reported as still open, instead of being reported closed.
|
|
20
|
+
- **Every element gets a ref.** `browser_snapshot`'s `maxNodes` caps only what is shown, so a ref the agent read from a long page still resolves.
|
|
21
|
+
- **`browser_page_info` reports the new page right after a navigation**, not the cached previous one.
|
|
22
|
+
- **`browser_web_search` resolves Google `/goto` result links**, and treats every Google TLD as Google's own, not a result.
|
|
23
|
+
- **The agent calls `browser_setup` before it has connected** instead of falling back to `curl`: the prompt now says the `browser_*` tools are available and the browser is just not connected yet.
|
|
24
|
+
|
|
25
|
+
## 0.13.0 — 2026-10-20
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
|
|
29
|
+
- **A killed daemon no longer crashes pi with `read ECONNRESET`.** The daemon's liveness probe, the client transport and the IPC server read the wire protocol through `node:readline`, which re-emits the socket's errors on its own interface with no listener. They now share a small line reader in `src/daemon/lines.ts` that only emits lines.
|
|
30
|
+
- **Waking from sleep no longer stacks up "Allow remote debugging" prompts.** Each dial to Chrome is one prompt, and the daemon abandoned a dial after 10s and started another on the next tool call, so an agent mid-task could open a dozen prompts before anyone was back at the keyboard. A dial now stays open for 5 minutes while callers get a "waiting for the Allow prompt" answer after 10s, every caller joins the dial in flight, and an outright failure starts a doubling cooldown (30s to 5min) during which no automatic dial happens. A dropped connection still re-dials once at the next demand. `/browser-setup` and `/browser-reload-daemon` send `Daemon.reconnect`, which bypasses the cooldown.
|
|
31
|
+
- **Tool failures now reach the model as errors.** `defineBrowserTool` set `isError` on the returned result, but pi only flags a result as an error when `execute` rejects, so every failed browser tool read as a success. Failed handlers now throw with the rendered error text; replayed errors still render in the UI.
|
|
32
|
+
- **The daemon transport no longer exits mid-handshake.** The connect timer was cleared on TCP connect, before the daemon's `registered` reply, and the socket was already unref'd, so a print-mode run could exit before registering. The timer now stays armed until the handshake settles; a daemon that accepts but never registers times out instead of hanging.
|
|
33
|
+
- **Disconnected clients are cleaned up in the daemon.** The idle-timer registration overwrote the bridge's disconnect handler, so router ownership and armed timers leaked per client. The IPC server now keeps a handler list; `deregister` is handled.
|
|
34
|
+
- **`--browser-namespace` works.** The flag was read before it was registered, so it was always undefined.
|
|
35
|
+
- **CDP sessions are reused or detached.** Switching tabs re-attached every time and left the old session alive; destroying the current target left `current()` pointing at it for up to 30s. Sessions are now tracked per target, detached on re-attach, and cleared on `Target.targetDestroyed` / `Inspector.detached`.
|
|
36
|
+
- **Connect timeouts no longer kick every client off the daemon**, overlapping `connect()` calls share one handshake, the detach release reads the session from `params`, and the duplicate-clientId reply is flushed before the socket closes.
|
|
37
|
+
- **Time budgets in `browser_network_requests` and `browser_snapshot` actually bound the work** through a concurrency-limited runner that honours the abort signal.
|
|
38
|
+
- **`browser_handle_dialog` clears the recorded dialog**, so the next `browser_page_info` no longer reports an already-handled dialog.
|
|
39
|
+
- **`browser_press_key` sends virtual key codes for function and modifier keys.**
|
|
40
|
+
- **`browser_open_urls` applies the tab-title marker after the page loads** instead of to the document being replaced.
|
|
41
|
+
- **`browser_print_to_pdf`, screenshots and uploads return `io_error`** on unwritable or unreadable paths instead of throwing.
|
|
42
|
+
- **Element refs are released** (`Runtime.releaseObject`) on every ref path; `browser_read_page` and `browser_open_urls` detach the sessions they attach.
|
|
43
|
+
- sharp "missing" is classified correctly under ESM; `isPathWithin` accepts siblings starting with `..`; `BH_DEBUG_CLICKS=0` disables debug; Linux `chromium-browser` process names are matched after `ps` truncation; discovery probes stale `DevToolsActivePort` candidates concurrently under one deadline.
|
|
44
|
+
|
|
45
|
+
### Changed
|
|
46
|
+
|
|
47
|
+
- **Bad arguments return `invalid_args`** instead of `invalid_state`.
|
|
48
|
+
- **`browser_switch_tab` resolves targetId prefixes against owned tabs only**, matching `browser_close_tab`.
|
|
49
|
+
- **`browser_execute_js` details hold a bounded preview**; the full value lives in the text.
|
|
50
|
+
- **CDP calls require params when the method needs them**, and `Input.*`, `Emulation.setDeviceMetricsOverride`, `Page.printToPDF` and `Page.captureScreenshot` have real schemas.
|
|
51
|
+
- **Tool `details` are typed** via a TypeBox schema on the definition and validated on replay.
|
|
52
|
+
- **Page-side JS is TypeScript** under `src/domains/page/*.page.ts`, type-checked against the DOM lib and injected via `fn.toString()`. `src/domains/fill-engine.ts` is removed.
|
|
53
|
+
- The browser prompt's lane lists and the skill file's tool reference are generated from the registry (`npm run gen:skill`).
|
|
54
|
+
|
|
55
|
+
### Added
|
|
56
|
+
|
|
57
|
+
- **Other pi extensions can contribute browser tools** through `registerBrowserTools` (see README, Extending).
|
|
58
|
+
- Type-aware ESLint (`npm run lint`) and `npm run check` running typecheck, boundaries, lint and tests. The boundary scanner gained a `raw-js-template` rule.
|
|
59
|
+
- Injection seams on the daemon entry, extension factory, setup and profile binding; tests for every tool in `src/domains` and the previously untested util, profile and daemon modules (365 tests to 1017).
|
|
60
|
+
|
|
5
61
|
## 0.12.0 — 2026-10-19
|
|
6
62
|
|
|
7
63
|
### Changed
|
package/README.md
CHANGED
|
@@ -182,6 +182,8 @@ tabs or navigating the tab to another origin.
|
|
|
182
182
|
| Tool | Purpose |
|
|
183
183
|
|---|---|
|
|
184
184
|
| `browser_run_script` | Run a temporary script with raw CDP and full Node access. See [Extending](#extending). |
|
|
185
|
+
| `browser_save_recipe` | Save the browser task the agent just finished as a replayable recipe. See [Saved recipes](#saved-recipes). |
|
|
186
|
+
| `browser_run_recipe` | Replay a saved recipe with no model calls, or list recipes when called with no name. |
|
|
185
187
|
| `browser_setup` | Agent-callable setup. Idempotent — how the agent recovers from `not initialized` on its own. |
|
|
186
188
|
|
|
187
189
|
`browser_snapshot`, `browser_network_requests`, `browser_console`, `browser_execute_js`,
|
|
@@ -257,6 +259,49 @@ browser_press_key({ key: "T", modifiers: 10 }) // Ctrl+Shift+T
|
|
|
257
259
|
|
|
258
260
|
---
|
|
259
261
|
|
|
262
|
+
## Saved recipes
|
|
263
|
+
|
|
264
|
+
Every page-changing browser call is journaled to `~/.pi/agent/browser-harness/journal/<session>.jsonl`
|
|
265
|
+
(user-only permissions, pruned after 30 days). Password values are never written.
|
|
266
|
+
|
|
267
|
+
After a multi-step task the agent offers to save it. `browser_save_recipe` turns that task into a YAML
|
|
268
|
+
file under `~/.pi/agent/browser-recipes/<site>/<name>.yaml` (or `.pi/browser-recipes/` in the project
|
|
269
|
+
with `project: true`), replacing the values you name as params with `{{param}}`. You confirm the recipe
|
|
270
|
+
before it is written. Saving over a name bumps `version` and keeps the old file under `.history/`.
|
|
271
|
+
|
|
272
|
+
Run one yourself with `/browser-recipe vendor-invoices-export month=2026-10` (no model involved), or ask
|
|
273
|
+
the agent ("run the invoices recipe for October"), which calls `browser_run_recipe`. `/browser-recipe`
|
|
274
|
+
with no name lists recipes; names tab-complete. Each step finds its element by role and name, waits up to
|
|
275
|
+
5s for it, and checks that the page changed the way the recorded run did. A status or alert message the
|
|
276
|
+
recorded run saw counts when its first words match, so "Exported 43 rows" satisfies "Exported 42 rows".
|
|
277
|
+
|
|
278
|
+
When a step fails, the replay stops and hands the agent the step, what it expected and a fresh snapshot.
|
|
279
|
+
The agent finishes your task from that step first, then calls `browser_save_recipe` with `repair_step`,
|
|
280
|
+
which replaces that step onward with what it just did, bumps `version` and keeps the old file. A failed
|
|
281
|
+
`/browser-recipe` goes to the agent the same way. If the page asks you to sign in, the replay stops and the
|
|
282
|
+
agent asks you to sign in; it never types credentials.
|
|
283
|
+
|
|
284
|
+
A recipe in the project's `.pi/browser-recipes/` comes from whoever wrote the repo, so it needs your
|
|
285
|
+
approval before its first run: you see its file, site, the URLs it opens, the files it uploads and its
|
|
286
|
+
downloads. The approval holds for those exact file bytes (kept in
|
|
287
|
+
`~/.pi/agent/browser-harness/trusted-recipes.json`); if the file changes, you are asked again. Without a UI
|
|
288
|
+
an unapproved project recipe does not run.
|
|
289
|
+
|
|
290
|
+
Reads are kept too: a `browser_http_get`, or a `browser_read_page` of a URL, becomes a read step, and a
|
|
291
|
+
replay returns what each one fetched (`browser_run_recipe` in its result, `/browser-recipe` as a message in
|
|
292
|
+
the session that starts no model turn). A read made with request headers or a tab id is left out, since
|
|
293
|
+
headers can carry credentials and a tab id means nothing next run. A `browser_execute_js` that changed
|
|
294
|
+
nothing is left out as well, so a replay does not return what it read; read data with `browser_http_get`
|
|
295
|
+
or `browser_read_page` when the recipe should hand it back.
|
|
296
|
+
|
|
297
|
+
If the site shows Cloudflare's "Just a moment..." check, a replay waits up to 30s for it to clear; if it
|
|
298
|
+
does not, the replay stops and the agent asks you to complete the check in the browser.
|
|
299
|
+
|
|
300
|
+
A task that typed into a password field, or used a `browser_execute_js` that changed the page or
|
|
301
|
+
`browser_run_script`, cannot be saved.
|
|
302
|
+
|
|
303
|
+
---
|
|
304
|
+
|
|
260
305
|
## Concurrency and isolation
|
|
261
306
|
|
|
262
307
|
**Parallel reads, serialized writes.** Observation tools run concurrently with each other and with
|
|
@@ -303,6 +348,91 @@ Scripts must live under the OS temp dir, the cwd, or `BH_SCRIPT_DIR`, and every
|
|
|
303
348
|
mandatory timeout (60s default, 600s max). They land on disk first, so they're reviewable and
|
|
304
349
|
re-runnable.
|
|
305
350
|
|
|
351
|
+
### Contributing tools from another pi extension
|
|
352
|
+
|
|
353
|
+
Another extension can add its own `browser_*` tools to this harness. Contributed tools share the
|
|
354
|
+
same client, mutation mutex, tab ownership, and lane prompt as the built-ins, so a `serialized` tool
|
|
355
|
+
you add queues behind `browser_click` exactly as `browser_fill` does.
|
|
356
|
+
|
|
357
|
+
Define the tool with `defineBrowserTool` and hand it to `registerBrowserTools` at your extension's
|
|
358
|
+
load time, before pi emits `session_start`. The harness registers it with pi on the next
|
|
359
|
+
`session_start` alongside the built-ins.
|
|
360
|
+
|
|
361
|
+
```ts
|
|
362
|
+
// my-extension/src/index.ts
|
|
363
|
+
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
|
364
|
+
import { Type } from "typebox";
|
|
365
|
+
import { defineBrowserTool, registerBrowserTools, ok } from "pi-browser-harness/src/index.ts";
|
|
366
|
+
import { evalJs } from "pi-browser-harness/src/domains/cdp-call.ts";
|
|
367
|
+
|
|
368
|
+
const pageHeightTool = defineBrowserTool({
|
|
369
|
+
name: "browser_page_height",
|
|
370
|
+
label: "Browser Page Height",
|
|
371
|
+
description: "Report the document's full scroll height in CSS pixels.",
|
|
372
|
+
promptSnippet: "Measure the page's scroll height",
|
|
373
|
+
promptGuidelines: [],
|
|
374
|
+
parameters: Type.Object({}),
|
|
375
|
+
concurrency: "parallel",
|
|
376
|
+
handler: async (_args, { client }) => {
|
|
377
|
+
const height = await evalJs(client, "document.documentElement.scrollHeight");
|
|
378
|
+
if (!height.success) return height;
|
|
379
|
+
return ok({ text: `Scroll height: ${String(height.data)}px` });
|
|
380
|
+
},
|
|
381
|
+
});
|
|
382
|
+
|
|
383
|
+
export default function myExtension(_pi: ExtensionAPI): void {
|
|
384
|
+
const added = registerBrowserTools([pageHeightTool]);
|
|
385
|
+
if (!added.success) console.warn(added.error.message);
|
|
386
|
+
}
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
`registerBrowserTools` returns a `Result`. The whole batch is rejected, and nothing is added, when any
|
|
390
|
+
name is already taken by a built-in, an earlier contribution, or another tool in the same batch. Pick
|
|
391
|
+
`concurrency: "serialized"` for anything that changes page or tab state; `"parallel"` for reads.
|
|
392
|
+
`browserTools()` lists what the harness will register, built-ins plus contributions so far.
|
|
393
|
+
|
|
394
|
+
A tool that wants a custom result view declares a `details` schema next to `parameters`. The handler
|
|
395
|
+
then returns `details` of that shape, and `renderResult` receives them typed. pi stores details with
|
|
396
|
+
the session and replays them later; the harness validates replayed details against the schema first,
|
|
397
|
+
so a renderer never sees a stale shape from an older session (it falls back to the stored text instead).
|
|
398
|
+
Details must be plain JSON: no `Map`s, and omit a key rather than setting it to `undefined`.
|
|
399
|
+
|
|
400
|
+
```ts
|
|
401
|
+
import { Text } from "@mariozechner/pi-tui";
|
|
402
|
+
|
|
403
|
+
const PageHeightDetails = Type.Object({ height: Type.Number(), unit: Type.Literal("px") });
|
|
404
|
+
|
|
405
|
+
const pageHeightTool = defineBrowserTool({
|
|
406
|
+
// ...name, label, description, promptSnippet, promptGuidelines, parameters, concurrency as above
|
|
407
|
+
details: PageHeightDetails,
|
|
408
|
+
handler: async (_args, { client }) => {
|
|
409
|
+
const height = await evalJs(client, "document.documentElement.scrollHeight");
|
|
410
|
+
if (!height.success) return height;
|
|
411
|
+
const px = typeof height.data === "number" ? height.data : 0;
|
|
412
|
+
return ok({ text: `Scroll height: ${px}px`, details: { height: px, unit: "px" } });
|
|
413
|
+
},
|
|
414
|
+
renderResult: (result, _expanded, theme) =>
|
|
415
|
+
new Text(theme.fg("muted", `${result.details.height}${result.details.unit}`), 0, 0),
|
|
416
|
+
});
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
Tools without a `details` schema need nothing extra: they may still attach any JSON object for logs,
|
|
420
|
+
and their `renderResult` (if any) sees it untyped.
|
|
421
|
+
|
|
422
|
+
### Known limits
|
|
423
|
+
|
|
424
|
+
Two extension points are documented but not implemented; the escape hatches that exist today are
|
|
425
|
+
listed with each.
|
|
426
|
+
|
|
427
|
+
- **No per-tab extension slot.** `TabSession` in `src/cdp/session.ts` carries only the harness's own
|
|
428
|
+
per-tab state (session id, dialog, page-info cache). A contributed tool cannot stash its own state
|
|
429
|
+
on a tab. Keep per-tab state in your extension keyed by `targetId` from `browser_current_tab` or
|
|
430
|
+
`browser_list_tabs`, or compute it on demand with `browser_run_script` through `daemon.session(targetId)`.
|
|
431
|
+
- **Hardcoded browser list.** `src/profile/paths.ts` knows the user-data directories of a fixed set
|
|
432
|
+
of Chromium browsers, so `/browser-profile` cannot discover one that is not on that list. Point the
|
|
433
|
+
harness at any Chromium profile directory with `CHROME_USER_DATA_DIR`, or skip discovery entirely
|
|
434
|
+
by attaching to a running browser's DevTools endpoint with `BU_CDP_WS`.
|
|
435
|
+
|
|
306
436
|
---
|
|
307
437
|
|
|
308
438
|
## Commands, flags, and environment
|
|
@@ -313,6 +443,7 @@ re-runnable.
|
|
|
313
443
|
| `/browser-profile` | Pick which browser profile the agent works in. Remembered across sessions. |
|
|
314
444
|
| `/browser-status` | Connection state, selected profile, and current page. |
|
|
315
445
|
| `/browser-reload-daemon` | Force a client restart. Rarely needed — the transport reconnects on its own. |
|
|
446
|
+
| `/browser-recipe` | Replay a saved recipe: `/browser-recipe NAME key=value`. No name lists recipes. |
|
|
316
447
|
|
|
317
448
|
| Flag / variable | Effect |
|
|
318
449
|
|---|---|
|
|
@@ -393,6 +524,7 @@ path-restricted and timeout-bounded, but that is a blast-radius limit, not a san
|
|
|
393
524
|
## Contributing
|
|
394
525
|
|
|
395
526
|
See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, conventions, and the PR process.
|
|
527
|
+
`npm run check` runs the typecheck, the boundary scan, the type-aware ESLint pass and the test suite in that order; it is the one command a PR must pass.
|
|
396
528
|
Release history lives in [CHANGELOG.md](CHANGELOG.md).
|
|
397
529
|
|
|
398
530
|
## License
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-browser-harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"description": "Browser control extension for pi — navigate, click, type, screenshot, and extract data from real Chrome via CDP",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package",
|
|
@@ -49,18 +49,24 @@
|
|
|
49
49
|
"typebox": "*"
|
|
50
50
|
},
|
|
51
51
|
"devDependencies": {
|
|
52
|
+
"@eslint/js": "10.0.1",
|
|
52
53
|
"@mariozechner/pi-coding-agent": "*",
|
|
53
54
|
"@mariozechner/pi-tui": "*",
|
|
54
55
|
"@types/ws": "^8.18.1",
|
|
56
|
+
"eslint": "10.11.0",
|
|
55
57
|
"typebox": "*",
|
|
56
|
-
"typescript": "^5.0.0"
|
|
58
|
+
"typescript": "^5.0.0",
|
|
59
|
+
"typescript-eslint": "8.71.0"
|
|
57
60
|
},
|
|
58
61
|
"scripts": {
|
|
59
|
-
"typecheck": "tsc --noEmit && tsc -p tsconfig.test.json",
|
|
60
|
-
"test": "node --conditions=import --import tsx --test \"test/{profile,domains,cdp,daemon,util}/**/*-test.ts\"",
|
|
62
|
+
"typecheck": "tsc -b tsconfig.page.json && tsc --noEmit && tsc -p tsconfig.test.json",
|
|
63
|
+
"test": "node --conditions=import --import tsx --test \"test/{profile,domains,cdp,daemon,util,types,recipes}/**/*-test.ts\"",
|
|
61
64
|
"boundaries": "tsx scripts/check-boundaries.ts",
|
|
62
|
-
"
|
|
63
|
-
"
|
|
65
|
+
"gen:skill": "node --conditions=import --import tsx scripts/gen-skill.ts",
|
|
66
|
+
"check": "npm run typecheck && npm run boundaries && npm run lint && npm test",
|
|
67
|
+
"prepublishOnly": "npm run typecheck",
|
|
68
|
+
"lint": "eslint .",
|
|
69
|
+
"lint:fix": "eslint . --fix"
|
|
64
70
|
},
|
|
65
71
|
"publishConfig": {
|
|
66
72
|
"access": "public"
|
|
@@ -69,7 +75,9 @@
|
|
|
69
75
|
"sharp": "^0.33.0"
|
|
70
76
|
},
|
|
71
77
|
"dependencies": {
|
|
78
|
+
"acorn": "^8.19.0",
|
|
72
79
|
"tsx": "^4.22.4",
|
|
73
|
-
"ws": "^8.20.0"
|
|
80
|
+
"ws": "^8.20.0",
|
|
81
|
+
"yaml": "^2.8.3"
|
|
74
82
|
}
|
|
75
83
|
}
|
|
@@ -41,6 +41,63 @@ Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. No screenshot
|
|
|
41
41
|
|
|
42
42
|
`browser_web_search` and `browser_read_page` each run in their own isolated tab and never touch the user's current tab. Pair them for a research question: search for candidate URLs, then read the promising ones.
|
|
43
43
|
|
|
44
|
+
## Tool reference
|
|
45
|
+
|
|
46
|
+
Every tool, its concurrency lane, and a one-line summary. `serialized` tools queue through one
|
|
47
|
+
shared mutation lane; `parallel` tools may run alongside them. The decision tree above says which
|
|
48
|
+
to reach for first; the tool descriptions and parameter schemas carry per-tool detail.
|
|
49
|
+
|
|
50
|
+
<!-- tools:start -->
|
|
51
|
+
Generated from `src/registry.ts` by `npm run gen:skill`; do not edit by hand.
|
|
52
|
+
|
|
53
|
+
| Tool | Lane | Summary |
|
|
54
|
+
|---|---|---|
|
|
55
|
+
| `browser_clear_headers` | serialized | Clear extra request headers from the current owned tab |
|
|
56
|
+
| `browser_click` | serialized | Click an element by ref (preferred) or pixel coordinates |
|
|
57
|
+
| `browser_close_tab` | serialized | Close an owned tab |
|
|
58
|
+
| `browser_console` | parallel | Read JS errors / console output on the current tab |
|
|
59
|
+
| `browser_current_tab` | parallel | Get current tab info |
|
|
60
|
+
| `browser_dispatch_key` | serialized | Dispatch a DOM KeyboardEvent on an element by ref (preferred) or selector |
|
|
61
|
+
| `browser_download` | serialized | Configure download directory |
|
|
62
|
+
| `browser_drag_and_drop` | serialized | Drag and drop between two coordinates |
|
|
63
|
+
| `browser_execute_js` | parallel | Execute JS in the page — surgical DOM reads (preferred over screenshot) |
|
|
64
|
+
| `browser_fill` | serialized | Fill a form field by ref (preferred) or selector (works with React/Vue controlled inputs) |
|
|
65
|
+
| `browser_fill_form` | serialized | Fill many form fields at once by ref — React-safe, self-confirming |
|
|
66
|
+
| `browser_focus` | serialized | Focus an element by ref (preferred) or selector |
|
|
67
|
+
| `browser_go_back` | serialized | Go back one page |
|
|
68
|
+
| `browser_go_forward` | serialized | Go forward one page |
|
|
69
|
+
| `browser_handle_dialog` | serialized | Accept or dismiss a JS dialog |
|
|
70
|
+
| `browser_http_get` | parallel | HTTP GET (outside browser; for APIs/static pages) |
|
|
71
|
+
| `browser_list_tabs` | parallel | List browser tabs |
|
|
72
|
+
| `browser_navigate` | serialized | Navigate to a URL |
|
|
73
|
+
| `browser_network_requests` | parallel | Inspect recent network requests on the current page |
|
|
74
|
+
| `browser_new_tab` | serialized | Open a new tab |
|
|
75
|
+
| `browser_open_urls` | serialized | Open multiple URLs in new tabs (parallel) |
|
|
76
|
+
| `browser_page_info` | parallel | Get current page URL, title, viewport, and scroll position |
|
|
77
|
+
| `browser_press_key` | serialized | Press a key (Enter, Tab, Escape, arrows, or any character) |
|
|
78
|
+
| `browser_print_to_pdf` | serialized | Print the current page to PDF |
|
|
79
|
+
| `browser_read_page` | parallel | Read a page's main content as clean text |
|
|
80
|
+
| `browser_reload` | serialized | Reload the page |
|
|
81
|
+
| `browser_run_recipe` | parallel | Replay a saved browser recipe, or list recipes |
|
|
82
|
+
| `browser_run_script` | parallel | Run a temporary script with daemon + Node access (security-bounded) |
|
|
83
|
+
| `browser_save_recipe` | parallel | Save the last browser task as a replayable recipe |
|
|
84
|
+
| `browser_screenshot` | parallel | Capture a screenshot — visual verification only |
|
|
85
|
+
| `browser_scroll` | serialized | Scroll the page (deltaY positive=down, negative=up) |
|
|
86
|
+
| `browser_select_option` | serialized | Select an option in a native <select> by ref (preferred) or selector |
|
|
87
|
+
| `browser_set_checked` | serialized | Set a checkbox/radio checked state by ref |
|
|
88
|
+
| `browser_set_headers` | serialized | Set extra request headers on the current owned tab |
|
|
89
|
+
| `browser_setup` | parallel | Initialize browser connection |
|
|
90
|
+
| `browser_snapshot` | parallel | Get accessibility-tree snapshot with stable element refs (default for page inspection) |
|
|
91
|
+
| `browser_switch_tab` | serialized | Switch tabs |
|
|
92
|
+
| `browser_type` | serialized | Type text into the focused element |
|
|
93
|
+
| `browser_upload_file` | serialized | Upload a file to a file input by ref (preferred) or selector |
|
|
94
|
+
| `browser_viewport_resize` | serialized | Resize the viewport (responsive testing) |
|
|
95
|
+
| `browser_wait` | parallel | Wait N seconds |
|
|
96
|
+
| `browser_wait_for` | parallel | Wait for an element or text to appear (or an element to disappear) |
|
|
97
|
+
| `browser_wait_for_load` | serialized | Wait for the page to finish loading |
|
|
98
|
+
| `browser_web_search` | serialized | Search the web for a query |
|
|
99
|
+
<!-- tools:end -->
|
|
100
|
+
|
|
44
101
|
## Connection Setup
|
|
45
102
|
|
|
46
103
|
Browser control is **on-demand** — the daemon does NOT start automatically.
|
|
@@ -76,6 +133,36 @@ it by opening tabs elsewhere — a different profile means different accounts.
|
|
|
76
133
|
|
|
77
134
|
You're attached to the user's real Chrome — never launch your own. If auth is required, stop and ask the user. If `browser_page_info` returns a dialog, handle it first with `browser_handle_dialog`.
|
|
78
135
|
|
|
136
|
+
## Saved recipes
|
|
137
|
+
|
|
138
|
+
`browser_save_recipe` compiles your last browser task into a YAML recipe; `browser_run_recipe` replays one with no model calls. What a recipe keeps:
|
|
139
|
+
|
|
140
|
+
- **Page actions**: navigate, new_tab, click, fill, select_option, type, press_key, scroll, wait_for, close_tab and the other replayable tools. Each step finds its element by role and name and checks the page changed as it did when recorded.
|
|
141
|
+
- **Reads**: `browser_http_get` and `browser_read_page` with a `url`. A replay returns what each one fetched, so a recipe can hand back data, not only a page. A read with `headers` or a `targetId` is left out.
|
|
142
|
+
- **Left out**: a `browser_execute_js` that changed nothing. A replay does not return what it read, so when the user wants data back, fetch it with `browser_http_get` (an API, a JSON endpoint) or `browser_read_page` before saving.
|
|
143
|
+
- **Refused**: typing into a password field, a `browser_execute_js` that changed the page, `browser_run_script`.
|
|
144
|
+
|
|
145
|
+
Params replace values you pass: each value must appear, as written, in a step's args or element name (a URL, typed text, a selected label). Its URL forms count too: `{{topic|url}}`, `{{topic|plus}}`, `{{topic|underscore}}`. A value that only appeared in a left-out call cannot be a param.
|
|
146
|
+
|
|
147
|
+
The file format, for reading a recipe; repair it with `browser_save_recipe` `repair_step`, not by editing it, so the repair is replayed before it is written:
|
|
148
|
+
|
|
149
|
+
```yaml
|
|
150
|
+
name: npm-downloads
|
|
151
|
+
site: api.npmjs.org
|
|
152
|
+
version: 1
|
|
153
|
+
params:
|
|
154
|
+
package: { type: string, example: pi-browser-harness }
|
|
155
|
+
steps:
|
|
156
|
+
- navigate: { url: "https://www.npmjs.com/package/{{package}}" }
|
|
157
|
+
expect: { url: "https://www.npmjs.com/package/{{package}}" } # origin and path only
|
|
158
|
+
- http_get: { url: "https://api.npmjs.org/downloads/range/last-month/{{package}}" }
|
|
159
|
+
- click: { role: button, name: Export CSV, nth: 0 } # the element: role+name+nth, or { css: "#export" }
|
|
160
|
+
expect: { appeared: [{ role: status, name: Exported 42 rows }] }
|
|
161
|
+
recorded: { at: "2026-11-01T15:14:25.918Z", session: 01a197ab }
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
At a sign-in page or a bot check (Cloudflare's "Just a moment...", waited out for 30s first) the replay stops: ask the user to sign in or complete the check; never do either yourself.
|
|
165
|
+
|
|
79
166
|
## Downloads land in the user's context
|
|
80
167
|
|
|
81
168
|
`browser_download` calls `Browser.setDownloadBehavior` with no `browserContextId`, which Chrome applies to its **default browser context** — the same one the user's own tabs are in. The override redirects the user's manual downloads too, and it lives on the browser connection the daemon keeps open, so it outlives the tab and the pi session. Symptom the user reports: *"I download things in Chrome and nothing appears in Downloads."*
|
|
@@ -111,11 +198,11 @@ When a workflow repeats 3+ times or needs Node.js APIs, write a script to disk a
|
|
|
111
198
|
|
|
112
199
|
## Troubleshooting: "Could not start the browser daemon"
|
|
113
200
|
|
|
114
|
-
`browser_setup` spawns the daemon as `node <tsx cli> <pkg>/src/daemon/
|
|
201
|
+
`browser_setup` spawns the daemon as `node <tsx cli> <pkg>/src/daemon/main.ts` with `stdio: "ignore"` (tsx is resolved from the package directory upward, so in pi's hoisted install it sits beside the package rather than inside it). A crash before the socket binds is completely silent and all you see is `Could not start the browser daemon. Check /tmp/pi-browser-daemon.sock.`
|
|
115
202
|
|
|
116
203
|
Run it in the foreground to see the real error (it prints whether it reached Chrome):
|
|
117
204
|
|
|
118
|
-
cd <pkg> && node "$(node -p "require.resolve('tsx/cli')")" src/daemon/
|
|
205
|
+
cd <pkg> && node "$(node -p "require.resolve('tsx/cli')")" src/daemon/main.ts
|
|
119
206
|
|
|
120
207
|
Known cause (package 0.11.1): `Error: Cannot find module 'typebox'` at `src/cdp/discovery.ts`. The package declares `typebox` in `peerDependencies` and `devDependencies`, and pi installs packages with dev and peer dependencies omitted, so the tree has no physical copy. It still resolves inside the pi extension host, which maps `typebox` to pi's bundled copy, but not in the spawned daemon process, which resolves modules from the package directory upward. The daemon dies, no socket appears, and every browser tool reports `not_connected`.
|
|
121
208
|
|