pi-browser-harness 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes to pi-browser-harness will be documented in this file.
+
+## [0.1.0] - 2026-05-02
+
+### Added
+- Initial release of pi-browser-harness.
+- 20 browser control tools (`browser_navigate`, `browser_screenshot`, `browser_click`, `browser_type`, `browser_press_key`, `browser_scroll`, `browser_execute_js`, `browser_http_get`, `browser_new_tab`, `browser_open_urls`, `browser_switch_tab`, `browser_list_tabs`, `browser_current_tab`, `browser_page_info`, `browser_go_back`, `browser_go_forward`, `browser_reload`, `browser_wait`, `browser_wait_for_load`, `browser_handle_dialog`).
+- Self-extending harness: `list_dynamic_tools`, `register_tool`, `remove_tool` — the agent can write new browser tools at runtime.
+- Guided setup command (`/browser-setup`) with Chrome detection, automatic browser-harness installation via `uv` or `git clone`.
+- `/browser-status` and `/browser-reload-daemon` commands for daemon health monitoring.
+- `--browser-namespace` and `--browser-debug-clicks` CLI flags.
+- Session persistence for tab history and daemon namespace across reloads and branch navigation.
+- System prompt injection with browser usage guidance and common workflow patterns.
+- Custom TUI renderers for screenshots and tab listings.
+- Dialog detection and handling for JS `alert`/`confirm`/`prompt`/`beforeunload`.
+- Parallel URL opening via `browser_open_urls` with live progress streaming.
+- Output truncation with temp-file fallback for large JS evaluation and HTTP responses.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Pi Browser Harness Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,212 @@
+# pi-browser-harness
+
+Full browser control for pi agents. Navigate, click, type, screenshot, and extract data from your real Chrome — driven by your coding agent.
+
+---
+
+## Quick Start
+
+```bash
+# 1. Install
+pi install npm:pi-browser-harness
+
+# Or install from GitHub:
+# pi install git:github.com/amankumarsingh77/pi-browser-harness
+
+# 2. Enable Chrome remote debugging
+#    Open chrome://inspect/#remote-debugging in Chrome,
+#    tick the "Discover network targets" checkbox, click Allow.
+#    No Chrome relaunch needed.
+
+# 3. Connect pi to Chrome
+/browser-setup
+```
+
+That's it. Try:
+
+```
+> navigate to https://github.com and take a screenshot
+```
+
+### Requirements
+
+- pi (latest)
+- Node.js ≥ 22
+- Chrome / Chromium / Edge (any recent version)
+
+> If `chrome://inspect/#remote-debugging` isn't available, you can launch Chrome with `--remote-debugging-port=9222` instead. The harness will find it either way.
+
+---
+
+## What you can do
+
+### Navigation & tabs
+
+| Tool | What it does |
+|------|--------------|
+| `browser_navigate` | Go to a URL (reuses the attached tab) |
+| `browser_new_tab` | Open a new tab, optionally navigate |
+| `browser_open_urls` | Open many URLs in parallel tabs |
+| `browser_go_back` / `browser_go_forward` | History navigation |
+| `browser_reload` | Reload the current page |
+| `browser_list_tabs` / `browser_current_tab` | Discover tabs |
+| `browser_switch_tab` | Switch the active tab |
+| `browser_page_info` | URL, title, viewport, scroll, dialog state |
+
+### Interaction
+
+| Tool | What it does |
+|------|--------------|
+| `browser_click` | Click at viewport coordinates (works through iframes / shadow DOM) |
+| `browser_type` | Type into the focused element |
+| `browser_press_key` | Send a key with optional modifiers |
+| `browser_scroll` | Scroll by delta pixels |
+| `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt` |
+
+### Visual & data
+
+| Tool | What it does |
+|------|--------------|
+| `browser_screenshot` | Capture the current page as PNG |
+| `browser_execute_js` | Run JS in the page and return the result |
+| `browser_http_get` | Direct HTTP GET — bypasses the browser, much faster for APIs |
+| `browser_wait` / `browser_wait_for_load` | Sleep, or wait for `readyState === 'complete'` |
+
+### Extending the harness at runtime
+
+| Tool | What it does |
+|------|--------------|
+| `browser_run_script` | Execute a temporary script file with daemon access |
+
+---
+
+## Usage
+
+The core pattern is **screenshot → act → screenshot → verify.** Don't assume an action worked — look.
+
+### Navigate and screenshot
+
+```
+browser_navigate("https://github.com/browser-use/browser-harness")
+browser_wait_for_load()
+browser_screenshot()
+```
+
+### Fill a form
+
+```
+browser_screenshot()                  # see the form
+browser_click(400, 200)               # click the search input
+browser_type("pi browser harness")    # type
+browser_press_key("Enter")            # submit
+browser_wait_for_load()
+browser_screenshot()                  # verify
+```
+
+### Extract data
+
+```js
+// Fast: hit the API directly, no browser
+browser_http_get("https://api.github.com/repos/browser-use/browser-harness")
+
+// DOM scraping for rendered content
+browser_execute_js("document.querySelector('.price').innerText")
+
+// Structured arrays
+browser_execute_js(`
+  JSON.stringify(
+    Array.from(document.querySelectorAll('.result')).map(el => ({
+      title: el.querySelector('h3')?.textContent,
+      link:  el.querySelector('a')?.href,
+    }))
+  )
+`)
+```
+
+### Open multiple results in parallel
+
+```
+browser_navigate("https://google.com/search?q=chrome+devtools+protocol")
+browser_open_urls({ urls: [
+  "https://chromedevtools.github.io/devtools-protocol/",
+  "https://developer.chrome.com/docs/devtools",
+]})
+browser_list_tabs()
+browser_switch_tab({ targetId: "..." })
+```
+
+### SPAs
+
+`browser_wait_for_load` only checks `readyState`. For React/Vue/etc., poll for the actual element you want:
+
+```js
+browser_execute_js("!!document.querySelector('.loaded-content')")
+```
+
+### Keyboard modifiers
+
+`browser_press_key` takes a modifier bitfield: `Alt=1`, `Ctrl=2`, `Meta/Cmd=4`, `Shift=8`. Combine with `|`.
+
+```
+browser_press_key("c", { modifiers: 2 })   // Ctrl+C
+browser_press_key("v", { modifiers: 4 })   // Cmd+V
+browser_press_key("T", { modifiers: 10 })  // Ctrl+Shift+T
+```
+
+### Writing a temporary script
+
+When the built-ins aren't enough, write a script to disk and run it with `browser_run_script`.
+
+```js
+// 1. Write the script to disk
+write("/tmp/scrape-results.js", `
+  const data = await daemon.evaluateJS(\`
+    JSON.stringify(
+      Array.from(document.querySelectorAll('${params.selector}'))
+        .map(el => el.textContent.trim())
+    )
+  \`);
+  return {
+    content: [{ type: "text", text: data }],
+    details: { items: JSON.parse(data) },
+  };
+`)
+
+// 2. Execute it
+browser_run_script("/tmp/scrape-results.js", { selector: ".result" })
+```
+
+Scripts receive `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, plus the standard JS globals. The script is on disk — auditable and re-runnable.
+
+---
+
+## Commands
+
+| Command | Description |
+|---------|-------------|
+| `/browser-setup` | Connect pi to Chrome (run once) |
+| `/browser-status` | Show daemon health and current page |
+| `/browser-reload-daemon` | Restart the connection |
+
+---
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---------|-----|
+| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow, then retry. |
+| Connection fails after Chrome restart | Run `/browser-reload-daemon`. |
+| Page seems loaded but content is missing | SPA — `browser_wait_for_load` isn't enough. Poll with `browser_execute_js` for the element you need. |
+| A JS dialog is blocking actions | `browser_page_info` will report the dialog. Use `browser_handle_dialog` to accept or dismiss. |
+
+---
+
+## Security
+
+This extension drives your real Chrome. The agent can see open tabs, read page content, submit forms, and act inside authenticated sessions. `browser_run_script` evaluates JavaScript in the pi process with full `require` access — review any temporary scripts the agent writes to `/tmp/` before executing them.
+
+---
+
+## License
+
+MIT — see [LICENSE](LICENSE).

package/package.json

@@ -0,0 +1,68 @@
+{
+  "name": "pi-browser-harness",
+  "version": "0.1.0",
+  "description": "Browser control extension for pi — navigate, click, type, screenshot, and extract data from real Chrome via CDP",
+  "keywords": [
+    "pi-package",
+    "pi-extension",
+    "browser",
+    "cdp",
+    "chrome",
+    "web-automation",
+    "screenshot",
+    "scraping"
+  ],
+  "author": "Pi Browser Harness Contributors",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/amankumarsingh77/pi-browser-harness.git"
+  },
+  "bugs": {
+    "url": "https://github.com/amankumarsingh77/pi-browser-harness/issues"
+  },
+  "homepage": "https://github.com/amankumarsingh77/pi-browser-harness#readme",
+  "engines": {
+    "node": ">=22.0.0"
+  },
+  "files": [
+    "src/",
+    "skills/",
+    "assets/",
+    "tsconfig.json",
+    "LICENSE",
+    "README.md",
+    "CHANGELOG.md"
+  ],
+  "pi": {
+    "extensions": [
+      "./src/index.ts"
+    ],
+    "skills": [
+      "./skills"
+    ],
+    "image": "https://raw.githubusercontent.com/amankumarsingh77/pi-browser-harness/main/assets/hero.png"
+  },
+  "peerDependencies": {
+    "@mariozechner/pi-coding-agent": "*",
+    "@mariozechner/pi-tui": "*",
+    "typebox": "*"
+  },
+  "devDependencies": {
+    "@mariozechner/pi-coding-agent": "*",
+    "@mariozechner/pi-tui": "*",
+    "@types/ws": "^8.18.1",
+    "typebox": "*",
+    "typescript": "^5.0.0"
+  },
+  "scripts": {
+    "typecheck": "tsc --noEmit",
+    "prepublishOnly": "tsc --noEmit"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "ws": "^8.20.0"
+  }
+}

package/skills/pi-browser-harness/SKILL.md

@@ -0,0 +1,213 @@
+---
+name: pi-browser-harness
+description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome.
+---
+
+# pi-browser-harness
+
+Direct browser control via CDP. Gives pi agents full control of a real Chrome browser.
+Browser tools coexist with all standard pi tools (read, bash, edit, write, etc.).
+
+## Setup
+
+Run `/browser-setup` to install browser-harness and connect to Chrome. Chrome must be running
+with remote debugging enabled. Start Chrome with:
+
+```
+--remote-debugging-port=9222
+```
+
+Check status with `/browser-status`. Restart the daemon with `/browser-reload-daemon`.
+
+## Core Workflow
+
+```
+browser_screenshot() → browser_click(x, y) → browser_screenshot()
+```
+
+1. **Screenshot first** — see what's on screen before any interaction
+2. **Coordinate click** — read the target pixel from the screenshot, click at (x, y)
+3. **Screenshot again** — verify the action worked
+
+Compositor-level clicks (`Input.dispatchMouseEvent`) work through iframes, shadow DOM,
+and cross-origin content. No selectors needed.
+
+## Available Tools
+
+### Navigation
+- `browser_navigate` — navigate to a URL (creates new tab on first call, reuses current tab after)
+- `browser_new_tab` — open a new tab, optionally navigate (use for first navigation)
+- `browser_open_urls` — open multiple URLs in parallel tabs (use after search to open result links)
+- `browser_go_back` / `browser_go_forward` / `browser_reload` — history navigation
+- `browser_page_info` — get URL, title, viewport, scroll position, or dialog info
+- `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` — tab management
+
+### Interaction
+- `browser_click` — click at viewport CSS-pixel coordinates
+- `browser_type` — type text into focused element
+- `browser_press_key` — press a key (Enter, Tab, Escape, arrows, etc.)
+- `browser_scroll` — scroll the page at coordinates
+
+### Visual
+- `browser_screenshot` — capture PNG screenshot
+
+### Data Extraction
+- `browser_execute_js` — execute JavaScript and return result
+- `browser_http_get` — direct HTTP GET outside browser (10-50x faster for APIs)
+
+### Utility
+- `browser_wait` — wait N seconds
+- `browser_wait_for_load` — wait for document.readyState === 'complete'
+- `browser_handle_dialog` — accept or dismiss JS dialogs (alert/confirm/prompt)
+
+### Extending
+- `browser_run_script` — execute a temporary script file with daemon access
+
+## Pattern Reference
+
+### Navigation
+```
+browser_new_tab("https://example.com") → browser_wait_for_load() → browser_screenshot()
+```
+
+### Form Filling
+```
+browser_screenshot() → find input coordinates
+→ browser_click(x, y) → browser_type("text")
+→ browser_press_key("Tab") → browser_screenshot()
+```
+
+### Data Extraction
+```
+browser_execute_js("document.querySelector('.price').innerText")
+// or for APIs directly:
+browser_http_get("https://api.example.com/data")
+```
+
+### Scrolling
+```
+browser_screenshot() → browser_scroll({ deltaY: -500 }) → browser_screenshot()
+```
+
+### Research (search engine + browser_open_urls)
+```
+browser_new_tab("https://google.com/search?q=query") → search engine results
+browser_open_urls({urls: ["url1", "url2"]}) → open result pages in parallel tabs
+browser_list_tabs() → see all open tabs with targetIds
+browser_switch_tab({targetId: "..."}) → switch to a specific tab
+browser_wait_for_load() → wait for the page to render
+browser_screenshot() → visually inspect the page
+browser_execute_js("document.querySelector('.content').innerText") → extract text
+```
+
+## Temporary Scripts
+
+When the built-in tools aren't enough, write a temporary script to disk and execute it
+with `browser_run_script`. Scripts run in the harness process with direct access to
+the browser daemon and full Node.js APIs — no dynamic tool registration needed.
+
+```
+  ● agent: needs to scrape 47 paginated search result pages
+  │
+  ● write("/tmp/scrape-pages.js", "...script using daemon.cdp() and daemon.evaluateJS()...")
+  │
+  ● browser_run_script("/tmp/scrape-pages.js", { urls: [...] })
+  │
+  ✓ script executes and returns results
+```
+
+The script is written to disk, so it's auditable and re-runnable.
+
+**When to use scripts:**
+- You're about to do the same 3+ step sequence more than once
+- You need a capability the built-in tools don't provide (e.g., bulk CSV export, structured extraction, pagination)
+- You discover a domain-specific pattern that should be reusable
+- The alternative is a fragile multi-turn loop of screenshots + clicks
+
+**What NOT to do:**
+- Don't write scripts for one-off actions — use the built-in browser_* tools directly.
+- Don't try to call browser_* tools from within a script — sequence them as separate tool calls.
+
+**Script bindings:**
+- `params` — the arguments passed to browser_run_script
+- `daemon` — browser daemon (daemon.cdp(), daemon.evaluateJS(), daemon.getPageInfo(), etc.)
+- `require` — Node.js require() (use 'fs', 'path', 'crypto', etc.)
+- `signal` — AbortSignal for cancellation
+- `onUpdate` — progress callback: onUpdate({ content: [{ type: 'text', text: '...' }] })
+- `ctx` — ExtensionContext (ctx.cwd, ctx.signal, etc.)
+- `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`
+
+**Example — a script that extracts structured data:**
+```javascript
+// /tmp/extract-products.js
+const info = await daemon.getPageInfo();
+if ("dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message);
+const data = await daemon.evaluateJS(`
+  JSON.stringify(Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({
+    title: el.querySelector('${params.titleSelector}')?.textContent?.trim(),
+    price: el.querySelector('${params.priceSelector}')?.textContent?.trim(),
+  })))
+`);
+return { content: [{ type: 'text', text: data }], details: { raw: JSON.parse(data) } };
+```
+
+## What NOT To Do
+
+- **Don't launch your own browser** — you're connected to the user's real Chrome
+- **Don't type credentials** — if you hit an auth wall, stop and ask the user
+- **Don't assume page state** — screenshot to confirm
+- **Don't write pixel coordinates into documentation** — describe how to locate targets (selectors, text, aria-label)
+- **Don't use browser_navigate for first navigation** — use browser_new_tab to explicitly preserve the user's active tab (browser_navigate auto-creates a new tab on first call, but explicit is clearer)
+- **Don't ignore dialogs** — if browser_page_info returns a dialog, handle it first
+
+## Dialogs
+
+JS dialogs (alert, confirm, prompt, beforeunload) freeze the page's JS thread.
+If browser_page_info returns a dialog, handle it with browser_handle_dialog
+before any other browser action.
+
+```
+// Check page_info first — if it shows a dialog:
+browser_handle_dialog({ action: "accept" })       // accept/confirm
+browser_handle_dialog({ action: "dismiss" })       // dismiss/cancel
+browser_handle_dialog({ action: "accept", promptText: "hello" })  // prompt with text
+```
+
+## SPAs (React, Vue, etc.)
+
+`browser_wait_for_load()` only checks `document.readyState === "complete"`.
+Single-page apps often paint content after this. After wait_for_load, use
+`browser_execute_js` to check for specific elements:
+
+```
+browser_execute_js("!!document.querySelector('.loaded-content')")
+```
+
+## Keyboard Modifiers
+
+`browser_press_key` supports modifier bitfield:
+- 1 = Alt, 2 = Ctrl, 4 = Meta/Cmd, 8 = Shift
+- Combine: Ctrl+Shift = 2|8 = 10, Cmd+Shift = 4|8 = 12
+
+## Troubleshooting
+
+- **Daemon not starting**: Run `/browser-setup` to re-run the guided setup
+- **"DevToolsActivePort not found"**: Open chrome://inspect/#remote-debugging, tick checkbox, click Allow
+- **Stale session**: Run `/browser-reload-daemon` to restart the daemon
+- **Dialog blocking**: Run browser_page_info to detect it, then browser_handle_dialog
+- **Status check**: Run `/browser-status` to see daemon health and current page
+
+## Architecture
+
+```
+pi agent → pi-browser-harness (TypeScript)
+               │ CDP WebSocket
+               ▼
+            Chrome
+
+Temporary scripts:
+  write("/tmp/script.js") → browser_run_script("/tmp/script.js")
+      │                              │
+      ▼                              ▼
+  script on disk (auditable)    executed in harness process with daemon access
+```