pi-browser-harness 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -2,6 +2,33 @@
 
 All notable changes to pi-browser-harness will be documented in this file.
 
+## 0.6.0 — 2026-06-21
+
+### Added
+
+- **Per-tab isolation for multi-agent safety.** Each agent's browser session now operates in a dedicated Chrome window with tab ownership tracking. Tabs opened via `browser_open_urls` and `browser_new_tab` are automatically registered in the ownership registry and tagged with a 🟢 prefix in the document title for user visibility. `browser_list_tabs` scoped to `"owned"` (default) only shows the current session's tabs.
+- **Unix socket daemon with auto-reconnect.** The browser daemon now binds to a Unix-domain socket (`pi-browser.sock`) in the harness temp directory instead of a TCP port. The transport layer reconnects automatically when Chrome restarts — no manual `/browser-reload-daemon` needed.
+- **On-demand browser initialization.** Chrome is no longer launched eagerly at harness startup. The harness attaches lazily on the first browser tool call, reducing resource usage when the agent isn't using the browser.
+- **`browser_setup` as agent-callable tool.** The setup tool can now be called programmatically by agents, not just via slash command. Idempotent — safe to call when already connected.
+
+### Fixed
+
+- **Chrome detection on macOS** now matches by process-name substring instead of exact binary path comparison, fixing false negatives when Chrome is running from different installation paths (e.g. `/Applications/Google Chrome.app/` vs user-local copies).
+
+### Added
+
+- **`browser_console`** — new tool. Reads JS errors and console messages from the active tab via two CDP sources merged into one buffer: `Runtime.consoleAPICalled` (page `console.*` calls and uncaught exceptions) and `Log.entryAdded` (browser-level entries — CSP violations, mixed content, deprecations, network errors). Filters: `levels` (log/info/warn/error/debug), `textPattern` (substring; wrap in slashes for regex), `sinceMs`, `limit` (default 50, cap 500). Each record carries a monotonic `seq`; the response includes `nextCursor` so callers can pass `sinceSeq` to see only what's new since the previous drain — the cursor pattern that makes "what did this action cause?" answerable in one call. Buffer is page-scoped (cleared on tab switch) and bounded at 500 records; `bufferOverflowed` flag reports drops since the last drain. Stack traces (top 3 frames) are preserved for error/warn records. Per-arg cap of 2 KB prevents a single `console.log(hugeBlob)` from blowing the buffer.
+- **`Log` CDP domain enabled** alongside Page/DOM/Runtime/Network/Accessibility on every attach. (`Runtime` was already enabled, so `Runtime.consoleAPICalled` was reachable; this adds the missing domain for browser-level entries.)
+- **`browser_console` ships with a custom `renderResult`** following the established pattern — collapsed: header counts (`3 errors · 2 warnings · 47 logs`) + last 5 rows. Expanded: full list with stack traces inline as code fences. Appends `keyHint("app.tools.expand", ...)` so the binding label adapts to user remaps.
+
+### Internal
+
+- New pure module `src/cdp/console-buffer.ts` mirrors the `network-buffer.ts` pattern (insertion-ordered Map, FIFO eviction at capacity 500, overflow flag reset per drain). Wired into the existing single-consumer event loop in `src/cdp/session.ts`.
+
+### Known follow-ups
+
+- No tests added in this release. Verification is manual against real Chrome.
+
 ## 0.4.0 — 2026-05-06
 
 ### Added

package/README.md

@@ -2,7 +2,29 @@
 
 ![pi-browser-harness](https://raw.githubusercontent.com/amankumarsingh77/pi-browser-harness/main/assets/hero.png)
 
-Full browser control for pi agents. Navigate, click, type, screenshot, and extract data from your real Chrome — all driven through natural language via CDP.
+Full browser control for pi agents in **your real Chrome** — your sessions, your cookies, your tabs. Drives navigation, structured page reads, network capture, clicks, typing, screenshots, and arbitrary scripts via CDP.
+
+---
+
+## Why pi-browser-harness?
+
+| Capability | pi-browser-harness | Playwright MCP | Stagehand | Puppeteer MCP |
+|---|:---:|:---:|:---:|:---:|
+| **Drives your real Chrome** (logged-in sessions preserved) | ✅ | ❌ launches its own browser | ❌ | ❌ |
+| **Coordinate clicks** that work through iframes, shadow DOM, cross-origin | ✅ | ❌ selector-based | ❌ selector-based | ❌ selector-based |
+| **Inline TUI screenshot rendering** (Kitty/iTerm2/Ghostty/WezTerm) | ✅ | ❌ | ❌ | ❌ |
+| **Accessibility-tree snapshot with click coords `@(x,y)` per element** | ✅ | ✅ tree only, no coords | partial | ❌ |
+| **Network request capture** with filters + body capture | ✅ | ✅ post-hoc list | ❌ | ❌ |
+| **Parallel tool execution** with automatic mutation serialization | ✅ | ❌ | ❌ | ❌ |
+| **Temporary scripts** with daemon + full Node.js | ✅ | ❌ | ❌ | ❌ |
+| **Direct HTTP GET** outside the browser (10–50× faster for APIs) | ✅ | ❌ | ❌ | ❌ |
+| **Tab ownership isolation** — never touches the user's other tabs | ✅ | N/A | N/A | N/A |
+| **Pi-native** — no MCP/JSON-RPC overhead, no extra LLM API keys | ✅ | ❌ MCP roundtrip | ❌ external LLM | ❌ MCP roundtrip |
+| **TypeScript strict mode**, zero `any`, all CDP casts documented | ✅ | unknown | unknown | unknown |
+| Ctrl+O expand/collapse on tool output | ✅ | ❌ | ❌ | ❌ |
+| Compositor-level dispatch (works on every site, no flakey waits) | ✅ | ❌ | ❌ | ❌ |
+
+If you live in pi and you want an agent driving the same Chrome you're already signed into, this is the only one that fits.
 
 ---
 
@@ -29,84 +51,144 @@ pi install npm:pi-browser-harness
 
 ---
 
+## Tool hierarchy — read this first
+
+```
+What do you need to know?
+
+  ├─ Page structure / what's clickable / labels?
+  │     → browser_snapshot     (DEFAULT — AX tree with @(x,y) per interactive element)
+  │
+  ├─ A specific element's value / attribute / coords?
+  │     → browser_execute_js   (e.g. el.innerText, el.getBoundingClientRect())
+  │
+  ├─ Network behavior on the current page?
+  │     → browser_network_requests
+  │
+  └─ Visual rendering (layout / colors / chart drew correctly)?
+        → browser_screenshot   (LAST RESORT — pixels only)
+```
+
+Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. **No screenshot round-trip needed** to find click targets — the snapshot already has them.
+
+---
+
 ## Tools
 
+### Page inspection (use these by default)
+
 | Tool | Purpose |
 |------|---------|
-| `browser_screenshot` | Capture a PNG screenshot of the current page |
-| `browser_click` | Click at viewport coordinates |
-| `browser_type` | Type text into the focused element |
-| `browser_press_key` | Press a key (Enter, Tab, Escape, arrows, etc.) with optional modifiers |
-| `browser_scroll` | Scroll the page by delta pixels |
-| `browser_navigate` | Navigate to a URL |
-| `browser_new_tab` | Open a new tab, optionally navigate |
+| `browser_snapshot` | **Default for page inspection.** Returns the CDP accessibility tree with click coords `@(x,y)` for every interactive element. Optional `includeScreenshot:true`. |
+| `browser_execute_js` | Surgical DOM reads — element text, attributes, `getBoundingClientRect()`. Cheapest, most precise. |
+| `browser_network_requests` | List recent network requests on the current tab. Filter by url/method/status/type/recency; optional response-body capture. |
+| `browser_page_info` | URL, title, viewport, scroll position, or pending dialog. |
+| `browser_http_get` | Direct HTTP GET outside the browser — 10-50× faster for APIs. |
+
+### Visual (last resort)
+
+| Tool | Purpose |
+|------|---------|
+| `browser_screenshot` | Capture PNG/JPEG. Use only when you need to verify visual rendering. |
+
+### Navigation
+
+| Tool | Purpose |
+|------|---------|
+| `browser_navigate` / `browser_new_tab` | Navigate or open a tab |
 | `browser_open_urls` | Open multiple URLs in parallel tabs |
 | `browser_go_back` / `browser_go_forward` / `browser_reload` | History navigation |
-| `browser_page_info` | Get URL, title, viewport, scroll position, or dialog state |
-| `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` | Tab discovery and switching |
-| `browser_execute_js` | Run JavaScript in the page and return the result |
-| `browser_http_get` | Direct HTTP GET outside the browser — much faster for APIs |
+| `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` / `browser_close_tab` | Tab management (only tabs this session opened) |
+
+### Interaction
+
+| Tool | Purpose |
+|------|---------|
+| `browser_click` | Click at viewport coordinates (use `@(x,y)` from `browser_snapshot`) |
+| `browser_type` | Type text into the focused element |
+| `browser_press_key` | Press a key with optional modifiers |
+| `browser_scroll` | Scroll the page by delta pixels |
+
+### Utility
+
+| Tool | Purpose |
+|------|---------|
 | `browser_wait` / `browser_wait_for_load` | Sleep, or wait for `readyState === 'complete'` |
 | `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt` |
 | `browser_run_script` | Execute a temporary script with daemon and Node.js access |
 
----
+Three tools (`browser_snapshot`, `browser_network_requests`, `browser_execute_js`) ship custom TUI rendering with **Ctrl+O** (`app.tools.expand`) to toggle between compact and full output.
 
-## Core Patterns
+---
 
-The golden rule: **screenshot → act → screenshot → verify.** Never assume — always look.
+## Core patterns
 
-### Navigation
+### Page inspection
 
 ```
-browser_new_tab({ url: "https://example.com" })
-browser_wait_for_load()
-browser_screenshot()
+browser_snapshot()
+# → AX outline with @(x,y) per button/link/input
 ```
 
-### Form filling
+### Form filling (no screenshots)
 
 ```
-browser_screenshot()                   # see the form
-browser_click({ x: 400, y: 200 })      # click the search input
-browser_type({ text: "query" })        # type
-browser_press_key({ key: "Enter" })    # submit
+browser_snapshot()                    # find input @(x,y) and labels
+browser_click({ x, y })               # click using snapshot's @(x,y)
+browser_type({ text: "query" })
+browser_press_key({ key: "Enter" })
 browser_wait_for_load()
-browser_screenshot()                   # verify
+browser_snapshot()                    # verify next state
 ```
 
 ### Data extraction
 
-```
+```js
+// One value
 browser_execute_js({ expression: "document.querySelector('.price').innerText" })
 
+// Direct API call outside the browser
 browser_http_get({ url: "https://api.github.com/repos/amankumarsingh77/pi-browser-harness" })
 
 // Structured arrays
-browser_execute_js({ expression: `Array.from(document.querySelectorAll('.result')).map(el => ({
-  title: el.querySelector('h3')?.textContent,
-  link:  el.querySelector('a')?.href,
-}))` })
+browser_execute_js({ expression: `JSON.stringify(
+  Array.from(document.querySelectorAll('.result')).map(el => ({
+    title: el.querySelector('h3')?.textContent,
+    link:  el.querySelector('a')?.href,
+  }))
+)` })
+```
+
+### Network debugging
+
+```
+browser_navigate({ url: "https://app.example.com/feed" })
+browser_wait_for_load()
+browser_network_requests({
+  urlPattern: "/api/",
+  statusFilter: { min: 400 },
+  includeResponseBodies: true
+})
 ```
 
 ### Research workflow
 
 ```
 browser_navigate({ url: "https://google.com/search?q=..." })
-// read search results
 browser_open_urls({ urls: ["url1", "url2", "url3"] })
 browser_list_tabs()
 browser_switch_tab({ targetId: "..." })
-browser_screenshot()
+browser_wait_for_load()
+browser_snapshot()
 browser_execute_js({ expression: "document.querySelector('.content').innerText" })
 ```
 
-### Scrolling
+### Visual verification (only when pixels matter)
 
 ```
-browser_screenshot()
-browser_scroll({ deltaY: -500 })
-browser_screenshot()
+browser_click({ x, y })         # got coords from browser_snapshot
+browser_snapshot()              # confirm the form transitioned
+browser_screenshot()            # ONLY if you need to verify a chart/modal/CSS rendered correctly
 ```
 
 ### Keyboard modifiers
@@ -118,55 +200,65 @@ browser_screenshot()
 | Meta / Cmd | 4 |
 | Shift | 8 |
 
-Combine with bitwise OR:
 ```
 browser_press_key({ key: "c", modifiers: 2 })    // Ctrl+C
 browser_press_key({ key: "v", modifiers: 4 })    // Cmd+V
 browser_press_key({ key: "T", modifiers: 10 })   // Ctrl+Shift+T
 ```
 
-### SPAs (React, Vue, etc.)
+### Dialogs
 
-`browser_wait_for_load` only checks `readyState`. Poll for elements after:
+JS dialogs freeze the page. Check `browser_page_info` first — if it reports a dialog, handle it before anything else:
 
 ```
-browser_execute_js({ expression: "!!document.querySelector('.loaded-content')" })
+browser_handle_dialog({ action: "accept" })       // confirm
+browser_handle_dialog({ action: "dismiss" })       // cancel
+browser_handle_dialog({ action: "accept", promptText: "hello" })  // prompt
 ```
 
-### Dialogs
+---
+
+## Parallel execution
 
-JS dialogs (`alert` / `confirm` / `prompt`) freeze the page. Check `browser_page_info` first — if it reports a dialog, handle it before anything else:
+Observation tools run in parallel by default. Mutation tools (`click`, `type`, `scroll`, `navigate`, `switch_tab`, …) are automatically serialized through a shared mutex — emit them in the same turn and the harness FIFO-queues them.
 
 ```
-browser_handle_dialog({ action: "accept" })       // confirm
-browser_handle_dialog({ action: "dismiss" })       // cancel
-browser_handle_dialog({ action: "accept", promptText: "hello" })  // prompt
+# All three run concurrently
+browser_snapshot()
+browser_network_requests({ sinceMs: 5000 })
+browser_http_get({ url: "..." })
 ```
 
 ---
 
-## Temporary Scripts
+## Tab ownership
 
-When built-in tools aren't enough, write a script to disk and run it:
+The harness never touches tabs you didn't open through it. On first attach it spawns a dedicated Chrome window; subsequent `browser_new_tab` calls open inside that window. `browser_list_tabs` defaults to `scope:"owned"` (pass `scope:"all"` to see read-only listings of your other tabs); `browser_switch_tab` and `browser_close_tab` refuse non-owned tabs.
+
+---
+
+## Temporary scripts
+
+When the built-in tools aren't enough, write a script to disk and run it. Scripts get a `daemon` binding for direct CDP access — much faster than chaining tool calls.
 
 ```js
-// 1. Write the script
 write("/tmp/scrape-pages.js", `
   const results = [];
   for (const url of params.urls) {
-    await daemon.cdp("Page.navigate", { url });
+    await daemon.session().call("Page.navigate", { url });
     await new Promise(r => setTimeout(r, 2000));
-    const title = await daemon.evaluateJS("document.title");
+    const title = await daemon.evaluateJs("document.title");
     results.push({ url, title });
   }
   return { content: [{ type: "text", text: JSON.stringify(results, null, 2) }] };
 `)
 
-// 2. Execute it
 browser_run_script({ path: "/tmp/scrape-pages.js", params: { urls: [...] } })
 ```
 
-**Script bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`.
+**Bindings:** `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`.
+
+`daemon` exposes: `evaluateJs`, `pageInfo`, `listTabs`, `switchTab`, `newTab`, `current`, and `session(targetId?)` for raw CDP via `session.call` / `session.callOnTarget` / `session.callBrowser` / `session.takeDialog`.
 
 Scripts are written to disk — auditable and re-runnable.
 
@@ -186,8 +278,9 @@ Scripts are written to disk — auditable and re-runnable.
 
 - Don't launch your own browser — you're connected to the user's real Chrome
 - Don't type credentials — if you hit an auth wall, stop and ask
-- Don't assume a page action worked — screenshot to verify
-- Don't write pixel coordinates into documentation — describe selectors and text instead
+- Don't screenshot to understand the page — `browser_snapshot` is the default
+- Don't screenshot to find click coordinates — `browser_snapshot`'s `@(x,y)` is exact
+- Don't screenshot to read a value — `browser_execute_js` is one round-trip
 - Don't ignore dialogs — check `browser_page_info` first
 
 ---
@@ -211,9 +304,10 @@ Temporary scripts run inside the harness process with full daemon and Node.js ac
 |---------|-----|
 | `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow |
 | Connection fails after Chrome restart | Run `/browser-reload-daemon` |
-| Page seems loaded but content is missing | SPA — poll with `browser_execute_js` for the element you need |
-| A JS dialog is blocking actions | `browser_page_info` will report it — use `browser_handle_dialog` |
+| Page seems loaded but content is missing | SPA — call `browser_snapshot` again, or `browser_execute_js` for a specific element |
+| JS dialog is blocking actions | `browser_page_info` will report it — use `browser_handle_dialog` |
 | Daemon not starting | Run `/browser-setup` to re-run guided setup |
+| Snapshot didn't return `@(x,y)` for a target | Element wasn't recognized as interactive. Fall back to `browser_execute_js` with `getBoundingClientRect()` |
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-browser-harness",
-  "version": "0.4.0",
+  "version": "0.6.0",
   "description": "Browser control extension for pi — navigate, click, type, screenshot, and extract data from real Chrome via CDP",
   "keywords": [
     "pi-package",
@@ -52,6 +52,7 @@
     "@mariozechner/pi-coding-agent": "*",
     "@mariozechner/pi-tui": "*",
     "@types/ws": "^8.18.1",
+    "tsx": "^4.22.4",
     "typebox": "*",
     "typescript": "^5.0.0"
   },

package/skills/pi-browser-harness/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: pi-browser-harness
-description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome. Default to browser_snapshot for understanding pages and browser_execute_js for surgical reads — browser_screenshot is for visual verification only.
+description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome. Default to browser_snapshot for understanding pages and browser_execute_js for surgical reads — browser_screenshot is for visual verification only.
 ---
 
 # pi-browser-harness
@@ -21,16 +21,39 @@ What do you need to know?
   ├─ Network behavior on the current page?
   │     → browser_network_requests
   │
+  ├─ JS errors / why did nothing happen after an action?
+  │     → browser_console     (DIAGNOSTIC — only when something looks broken)
+  │
   └─ Visual rendering (layout / colors / chart drew correctly)?
         → browser_screenshot   (LAST RESORT — pixels only)
 ```
 
 Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. No screenshot round-trip.
 
+## Connection Setup
+
+Browser control is **on-demand** — the daemon does NOT start automatically.
+If you try a browser tool and get a `not_connected` error, tell the user to
+run `/browser-setup` first. This opens the daemon and connects to Chrome.
+Once initialized, all subsequent sessions reuse the same connection silently.
+
+**Before calling any browser tool**, the runtime checks for the daemon socket
+at `/tmp/pi-browser-daemon.sock`. If the socket is missing (user hasn't run
+`/browser-setup`), you get: `"Browser harness not initialized. Run /browser-setup first"`.
+
+**Do not ask the user.** Call `browser_setup` directly — it spawns the daemon,
+connects to Chrome, and opens a test tab. The user sees a single "Allow Remote
+Debugging" prompt the first time. After that, all sessions reuse the same connection.
+`browser_setup` is idempotent — safe to call even when already connected.
+
 ## Connection
 
 You're attached to the user's real Chrome — never launch your own. If auth is required, stop and ask the user. If `browser_page_info` returns a dialog, handle it first with `browser_handle_dialog`.
 
+## Diagnosing a "nothing happened" moment
+
+When an action runs but the page didn't change, capture `browser_console`'s `nextCursor` *before* the action, take the action, then call `browser_console({ sinceSeq: <cursor> })` after — this isolates what your action caused from what was already there. Pair with `browser_network_requests({ sinceMs: 5000 })` to see if an API call fired and failed. The console buffer is page-scoped: it clears on tab switch, capacity 500.
+
 ## Temporary scripts
 
 When a workflow repeats 3+ times or needs Node.js APIs, write a script to disk and run it with `browser_run_script`. Scripts get a `daemon` binding for direct CDP access — much faster than chaining tool calls.