pi-browser-harness 0.3.2 → 0.5.0

package/CHANGELOG.md

@@ -2,6 +2,51 @@
 
 All notable changes to pi-browser-harness will be documented in this file.
 
+## 0.5.0 — 2026-05-07
+
+### Added
+
+- **`browser_console`** — new tool. Reads JS errors and console messages from the active tab via two CDP sources merged into one buffer: `Runtime.consoleAPICalled` (page `console.*` calls and uncaught exceptions) and `Log.entryAdded` (browser-level entries — CSP violations, mixed content, deprecations, network errors). Filters: `levels` (log/info/warn/error/debug), `textPattern` (substring; wrap in slashes for regex), `sinceMs`, `limit` (default 50, cap 500). Each record carries a monotonic `seq`; the response includes `nextCursor` so callers can pass `sinceSeq` to see only what's new since the previous drain — the cursor pattern that makes "what did this action cause?" answerable in one call. Buffer is page-scoped (cleared on tab switch) and bounded at 500 records; `bufferOverflowed` flag reports drops since the last drain. Stack traces (top 3 frames) are preserved for error/warn records. Per-arg cap of 2 KB prevents a single `console.log(hugeBlob)` from blowing the buffer.
+- **`Log` CDP domain enabled** alongside Page/DOM/Runtime/Network/Accessibility on every attach. (`Runtime` was already enabled, so `Runtime.consoleAPICalled` was reachable; this adds the missing domain for browser-level entries.)
+- **`browser_console` ships with a custom `renderResult`** following the established pattern — collapsed: header counts (`3 errors · 2 warnings · 47 logs`) + last 5 rows. Expanded: full list with stack traces inline as code fences. Appends `keyHint("app.tools.expand", ...)` so the binding label adapts to user remaps.
+
+### Internal
+
+- New pure module `src/cdp/console-buffer.ts` mirrors the `network-buffer.ts` pattern (insertion-ordered Map, FIFO eviction at capacity 500, overflow flag reset per drain). Wired into the existing single-consumer event loop in `src/cdp/session.ts`.
+
+### Known follow-ups
+
+- No tests added in this release. Verification is manual against real Chrome.
+
+## 0.4.0 — 2026-05-06
+
+### Added
+
+- **`browser_snapshot`** — new tool. Returns the structured CDP accessibility tree (roles, names, states, hierarchy) for the current page. For every interactive element (button, link, textbox, checkbox, etc.) the outline includes click coordinates as `@(x,y)`, fetched via `DOM.getBoxModel` per node in parallel under a 1.5s aggregate budget. Pass these straight to `browser_click` — no `browser_screenshot` round-trip needed. `format:"json"` returns the slim structure with `box: {x,y,width,height,cx,cy}` per node. Optional `includeScreenshot:true` attaches a JPEG (q=80) when visual confirmation is also wanted.
+- **`browser_network_requests`** — new tool, replacing the deprecated `browser_get_network_log` placeholder. Lists requests captured on the current tab since attach with filters: `urlPattern` (substring; wrap in slashes for regex), `methodFilter`, `statusFilter`, `resourceTypes`, `sinceMs`, `limit` (default 50, cap 500). `includeResponseBodies:true` fetches `Network.getResponseBody` per matched record under a 5s aggregate budget with a 50 KB per-body cap. Buffer is page-scoped (cleared on tab switch) and bounded at 500 records; `bufferOverflowed` flag in the result reports drops since the last drain.
+- **Tab ownership / harness-window isolation.** New `OwnershipRegistry` tracks which page targets this session opened. The harness now creates a dedicated Chrome window on first attach (`newWindow:true`) instead of grabbing the user's foreground tab; subsequent `browser_new_tab` calls open inside that window via `openerId`. `browser_list_tabs` defaults to `scope:"owned"`; pass `scope:"all"` to see the user's other tabs read-only. `browser_switch_tab` and the new **`browser_close_tab`** refuse non-owned tabs with a clear remediation hint. Ownership is persisted across session reloads via `BrowserState`. The session also subscribes `Target.setDiscoverTargets` and reaps `targetDestroyed` events so the registry stays in sync.
+- **`Accessibility` CDP domain enabled** alongside Page/DOM/Runtime/Network on every attach.
+- **`Ctrl+O` (`app.tools.expand`) expand/collapse on tool output.** Three tools now ship custom `renderResult`:
+  - `browser_snapshot` — collapsed: 4-line summary (node count, URL, landmarks/buttons/inputs, screenshot status). Expanded: full indented outline + inline screenshot when `includeScreenshot:true`.
+  - `browser_network_requests` — collapsed: header + first 5 rows. Expanded: full markdown table + per-request body sections when bodies were requested.
+  - `browser_execute_js` — collapsed: size + first 120-char preview. Expanded: pretty-printed JSON if value parses, otherwise raw value, in a code fence.
+  All three append a `keyHint("app.tools.expand", ...)` so the binding label adapts to user remaps.
+
+### Changed
+
+- **Tool prompts pivoted from screenshot-first to snapshot-first.** `browser_screenshot`'s description now explicitly says "NOT a default exploration tool". `browser_snapshot` is documented as the default for understanding pages; `browser_execute_js` as the default for surgical reads. `browser_click` guidance no longer instructs the agent to screenshot for coordinates — it points at `browser_snapshot`'s `@(x,y)` hints. `browser_open_urls` post-step flipped from screenshot to snapshot.
+- **`SKILL.md` rewritten** (~342 lines → ~53 lines). Frontmatter description carries the tool-hierarchy hint (always in context per pi's progressive-disclosure model). Body keeps only what isn't already in tool prompts: the decision tree, the connection rules (real Chrome, no creds, dialog-first), and the `browser_run_script` daemon bindings (the only tool whose API can't be inferred from its prompt). Pattern reference, parallelization details, troubleshooting, and tool enumeration removed as duplication.
+- **CDP `Network.*` events** are now consumed by an in-process aggregator (`src/cdp/network-buffer.ts`) wired into the existing single-consumer event loop in `src/cdp/session.ts`. Pure module, ring-buffered, non-destructive drain.
+
+### Removed
+
+- **`browser_get_network_log`** — was a deprecated placeholder in v0.3 that returned a "use PerformanceObserver" message because the CDP event stream had no public drain API. Replaced by `browser_network_requests`.
+
+### Known follow-ups
+
+- Live network streaming (`browser_network_monitor` real-time during a click) — deferred; reuses the same buffer plumbing once the post-hoc form is proven.
+- No tests added in this release. Verification is manual against real Chrome.
+
 ## 0.3.2 — 2026-05-05
 
 ### Added

package/README.md

@@ -2,7 +2,29 @@
 
 ![pi-browser-harness](https://raw.githubusercontent.com/amankumarsingh77/pi-browser-harness/main/assets/hero.png)
 
-Full browser control for pi agents. Navigate, click, type, screenshot, and extract data from your real Chrome — all driven through natural language via CDP.
+Full browser control for pi agents in **your real Chrome** — your sessions, your cookies, your tabs. Drives navigation, structured page reads, network capture, clicks, typing, screenshots, and arbitrary scripts via CDP.
+
+---
+
+## Why pi-browser-harness?
+
+| Capability | pi-browser-harness | Playwright MCP | Stagehand | Puppeteer MCP |
+|---|:---:|:---:|:---:|:---:|
+| **Drives your real Chrome** (logged-in sessions preserved) | ✅ | ❌ launches its own browser | ❌ | ❌ |
+| **Coordinate clicks** that work through iframes, shadow DOM, cross-origin | ✅ | ❌ selector-based | ❌ selector-based | ❌ selector-based |
+| **Inline TUI screenshot rendering** (Kitty/iTerm2/Ghostty/WezTerm) | ✅ | ❌ | ❌ | ❌ |
+| **Accessibility-tree snapshot with click coords `@(x,y)` per element** | ✅ | ✅ tree only, no coords | partial | ❌ |
+| **Network request capture** with filters + body capture | ✅ | ✅ post-hoc list | ❌ | ❌ |
+| **Parallel tool execution** with automatic mutation serialization | ✅ | ❌ | ❌ | ❌ |
+| **Temporary scripts** with daemon + full Node.js | ✅ | ❌ | ❌ | ❌ |
+| **Direct HTTP GET** outside the browser (10–50× faster for APIs) | ✅ | ❌ | ❌ | ❌ |
+| **Tab ownership isolation** — never touches the user's other tabs | ✅ | N/A | N/A | N/A |
+| **Pi-native** — no MCP/JSON-RPC overhead, no extra LLM API keys | ✅ | ❌ MCP roundtrip | ❌ external LLM | ❌ MCP roundtrip |
+| **TypeScript strict mode**, zero `any`, all CDP casts documented | ✅ | unknown | unknown | unknown |
+| Ctrl+O expand/collapse on tool output | ✅ | ❌ | ❌ | ❌ |
+| Compositor-level dispatch (works on every site, no flakey waits) | ✅ | ❌ | ❌ | ❌ |
+
+If you live in pi and you want an agent driving the same Chrome you're already signed into, this is the only one that fits.
 
 ---
 
@@ -29,84 +51,144 @@ pi install npm:pi-browser-harness
 
 ---
 
+## Tool hierarchy — read this first
+
+```
+What do you need to know?
+
+  ├─ Page structure / what's clickable / labels?
+  │     → browser_snapshot     (DEFAULT — AX tree with @(x,y) per interactive element)
+  │
+  ├─ A specific element's value / attribute / coords?
+  │     → browser_execute_js   (e.g. el.innerText, el.getBoundingClientRect())
+  │
+  ├─ Network behavior on the current page?
+  │     → browser_network_requests
+  │
+  └─ Visual rendering (layout / colors / chart drew correctly)?
+        → browser_screenshot   (LAST RESORT — pixels only)
+```
+
+Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. **No screenshot round-trip needed** to find click targets — the snapshot already has them.
+
+---
+
 ## Tools
 
+### Page inspection (use these by default)
+
 | Tool | Purpose |
 |------|---------|
-| `browser_screenshot` | Capture a PNG screenshot of the current page |
-| `browser_click` | Click at viewport coordinates |
-| `browser_type` | Type text into the focused element |
-| `browser_press_key` | Press a key (Enter, Tab, Escape, arrows, etc.) with optional modifiers |
-| `browser_scroll` | Scroll the page by delta pixels |
-| `browser_navigate` | Navigate to a URL |
-| `browser_new_tab` | Open a new tab, optionally navigate |
+| `browser_snapshot` | **Default for page inspection.** Returns the CDP accessibility tree with click coords `@(x,y)` for every interactive element. Optional `includeScreenshot:true`. |
+| `browser_execute_js` | Surgical DOM reads — element text, attributes, `getBoundingClientRect()`. Cheapest, most precise. |
+| `browser_network_requests` | List recent network requests on the current tab. Filter by url/method/status/type/recency; optional response-body capture. |
+| `browser_page_info` | URL, title, viewport, scroll position, or pending dialog. |
+| `browser_http_get` | Direct HTTP GET outside the browser — 10-50× faster for APIs. |
+
+### Visual (last resort)
+
+| Tool | Purpose |
+|------|---------|
+| `browser_screenshot` | Capture PNG/JPEG. Use only when you need to verify visual rendering. |
+
+### Navigation
+
+| Tool | Purpose |
+|------|---------|
+| `browser_navigate` / `browser_new_tab` | Navigate or open a tab |
 | `browser_open_urls` | Open multiple URLs in parallel tabs |
 | `browser_go_back` / `browser_go_forward` / `browser_reload` | History navigation |
-| `browser_page_info` | Get URL, title, viewport, scroll position, or dialog state |
-| `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` | Tab discovery and switching |
-| `browser_execute_js` | Run JavaScript in the page and return the result |
-| `browser_http_get` | Direct HTTP GET outside the browser — much faster for APIs |
+| `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` / `browser_close_tab` | Tab management (only tabs this session opened) |
+
+### Interaction
+
+| Tool | Purpose |
+|------|---------|
+| `browser_click` | Click at viewport coordinates (use `@(x,y)` from `browser_snapshot`) |
+| `browser_type` | Type text into the focused element |
+| `browser_press_key` | Press a key with optional modifiers |
+| `browser_scroll` | Scroll the page by delta pixels |
+
+### Utility
+
+| Tool | Purpose |
+|------|---------|
 | `browser_wait` / `browser_wait_for_load` | Sleep, or wait for `readyState === 'complete'` |
 | `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt` |
 | `browser_run_script` | Execute a temporary script with daemon and Node.js access |
 
----
+Three tools (`browser_snapshot`, `browser_network_requests`, `browser_execute_js`) ship custom TUI rendering with **Ctrl+O** (`app.tools.expand`) to toggle between compact and full output.
 
-## Core Patterns
+---
 
-The golden rule: **screenshot → act → screenshot → verify.** Never assume — always look.
+## Core patterns
 
-### Navigation
+### Page inspection
 
 ```
-browser_new_tab({ url: "https://example.com" })
-browser_wait_for_load()
-browser_screenshot()
+browser_snapshot()
+# → AX outline with @(x,y) per button/link/input
 ```
 
-### Form filling
+### Form filling (no screenshots)
 
 ```
-browser_screenshot()                   # see the form
-browser_click({ x: 400, y: 200 })      # click the search input
-browser_type({ text: "query" })        # type
-browser_press_key({ key: "Enter" })    # submit
+browser_snapshot()                    # find input @(x,y) and labels
+browser_click({ x, y })               # click using snapshot's @(x,y)
+browser_type({ text: "query" })
+browser_press_key({ key: "Enter" })
 browser_wait_for_load()
-browser_screenshot()                   # verify
+browser_snapshot()                    # verify next state
 ```
 
 ### Data extraction
 
-```
+```js
+// One value
 browser_execute_js({ expression: "document.querySelector('.price').innerText" })
 
+// Direct API call outside the browser
 browser_http_get({ url: "https://api.github.com/repos/amankumarsingh77/pi-browser-harness" })
 
 // Structured arrays
-browser_execute_js({ expression: `Array.from(document.querySelectorAll('.result')).map(el => ({
-  title: el.querySelector('h3')?.textContent,
-  link:  el.querySelector('a')?.href,
-}))` })
+browser_execute_js({ expression: `JSON.stringify(
+  Array.from(document.querySelectorAll('.result')).map(el => ({
+    title: el.querySelector('h3')?.textContent,
+    link:  el.querySelector('a')?.href,
+  }))
+)` })
+```
+
+### Network debugging
+
+```
+browser_navigate({ url: "https://app.example.com/feed" })
+browser_wait_for_load()
+browser_network_requests({
+  urlPattern: "/api/",
+  statusFilter: { min: 400 },
+  includeResponseBodies: true
+})
 ```
 
 ### Research workflow
 
 ```
 browser_navigate({ url: "https://google.com/search?q=..." })
-// read search results
 browser_open_urls({ urls: ["url1", "url2", "url3"] })
 browser_list_tabs()
 browser_switch_tab({ targetId: "..." })
-browser_screenshot()
+browser_wait_for_load()
+browser_snapshot()
 browser_execute_js({ expression: "document.querySelector('.content').innerText" })
 ```
 
-### Scrolling
+### Visual verification (only when pixels matter)
 
 ```
-browser_screenshot()
-browser_scroll({ deltaY: -500 })
-browser_screenshot()
+browser_click({ x, y })         # got coords from browser_snapshot
+browser_snapshot()              # confirm the form transitioned
+browser_screenshot()            # ONLY if you need to verify a chart/modal/CSS rendered correctly
 ```
 
 ### Keyboard modifiers
@@ -118,55 +200,65 @@ browser_screenshot()
 | Meta / Cmd | 4 |
 | Shift | 8 |
 
-Combine with bitwise OR:
 ```
 browser_press_key({ key: "c", modifiers: 2 })    // Ctrl+C
 browser_press_key({ key: "v", modifiers: 4 })    // Cmd+V
 browser_press_key({ key: "T", modifiers: 10 })   // Ctrl+Shift+T
 ```
 
-### SPAs (React, Vue, etc.)
+### Dialogs
 
-`browser_wait_for_load` only checks `readyState`. Poll for elements after:
+JS dialogs freeze the page. Check `browser_page_info` first — if it reports a dialog, handle it before anything else:
 
 ```
-browser_execute_js({ expression: "!!document.querySelector('.loaded-content')" })
+browser_handle_dialog({ action: "accept" })       // confirm
+browser_handle_dialog({ action: "dismiss" })       // cancel
+browser_handle_dialog({ action: "accept", promptText: "hello" })  // prompt
 ```
 
-### Dialogs
+---
+
+## Parallel execution
 
-JS dialogs (`alert` / `confirm` / `prompt`) freeze the page. Check `browser_page_info` first — if it reports a dialog, handle it before anything else:
+Observation tools run in parallel by default. Mutation tools (`click`, `type`, `scroll`, `navigate`, `switch_tab`, …) are automatically serialized through a shared mutex — emit them in the same turn and the harness FIFO-queues them.
 
 ```
-browser_handle_dialog({ action: "accept" })       // confirm
-browser_handle_dialog({ action: "dismiss" })       // cancel
-browser_handle_dialog({ action: "accept", promptText: "hello" })  // prompt
+# All three run concurrently
+browser_snapshot()
+browser_network_requests({ sinceMs: 5000 })
+browser_http_get({ url: "..." })
 ```
 
 ---
 
-## Temporary Scripts
+## Tab ownership
 
-When built-in tools aren't enough, write a script to disk and run it:
+The harness never touches tabs you didn't open through it. On first attach it spawns a dedicated Chrome window; subsequent `browser_new_tab` calls open inside that window. `browser_list_tabs` defaults to `scope:"owned"` (pass `scope:"all"` to see read-only listings of your other tabs); `browser_switch_tab` and `browser_close_tab` refuse non-owned tabs.
+
+---
+
+## Temporary scripts
+
+When the built-in tools aren't enough, write a script to disk and run it. Scripts get a `daemon` binding for direct CDP access — much faster than chaining tool calls.
 
 ```js
-// 1. Write the script
 write("/tmp/scrape-pages.js", `
   const results = [];
   for (const url of params.urls) {
-    await daemon.cdp("Page.navigate", { url });
+    await daemon.session().call("Page.navigate", { url });
     await new Promise(r => setTimeout(r, 2000));
-    const title = await daemon.evaluateJS("document.title");
+    const title = await daemon.evaluateJs("document.title");
     results.push({ url, title });
   }
   return { content: [{ type: "text", text: JSON.stringify(results, null, 2) }] };
 `)
 
-// 2. Execute it
 browser_run_script({ path: "/tmp/scrape-pages.js", params: { urls: [...] } })
 ```
 
-**Script bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`.
+**Bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`.
+
+`daemon` exposes: `evaluateJs`, `pageInfo`, `listTabs`, `switchTab`, `newTab`, `current`, and `session(targetId?)` for raw CDP via `session.call` / `session.callOnTarget` / `session.callBrowser` / `session.takeDialog`.
 
 Scripts are written to disk — auditable and re-runnable.
 
@@ -186,8 +278,9 @@ Scripts are written to disk — auditable and re-runnable.
 
 - Don't launch your own browser — you're connected to the user's real Chrome
 - Don't type credentials — if you hit an auth wall, stop and ask
-- Don't assume a page action worked — screenshot to verify
-- Don't write pixel coordinates into documentation — describe selectors and text instead
+- Don't screenshot to understand the page — `browser_snapshot` is the default
+- Don't screenshot to find click coordinates — `browser_snapshot`'s `@(x,y)` is exact
+- Don't screenshot to read a value — `browser_execute_js` is one round-trip
 - Don't ignore dialogs — check `browser_page_info` first
 
 ---
@@ -211,9 +304,10 @@ Temporary scripts run inside the harness process with full daemon and Node.js ac
 |---------|-----|
 | `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow |
 | Connection fails after Chrome restart | Run `/browser-reload-daemon` |
-| Page seems loaded but content is missing | SPA — poll with `browser_execute_js` for the element you need |
-| A JS dialog is blocking actions | `browser_page_info` will report it — use `browser_handle_dialog` |
+| Page seems loaded but content is missing | SPA — call `browser_snapshot` again, or `browser_execute_js` for a specific element |
+| JS dialog is blocking actions | `browser_page_info` will report it — use `browser_handle_dialog` |
 | Daemon not starting | Run `/browser-setup` to re-run guided setup |
+| Snapshot didn't return `@(x,y)` for a target | Element wasn't recognized as interactive. Fall back to `browser_execute_js` with `getBoundingClientRect()` |
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-browser-harness",
-  "version": "0.3.2",
+  "version": "0.5.0",
   "description": "Browser control extension for pi — navigate, click, type, screenshot, and extract data from real Chrome via CDP",
   "keywords": [
     "pi-package",