pi-browser-harness 0.3.2 → 0.5.0

package/skills/pi-browser-harness/SKILL.md

@@ -1,271 +1,60 @@
 ---
 name: pi-browser-harness
-description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome.
+description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome. Default to browser_snapshot for understanding pages and browser_execute_js for surgical reads — browser_screenshot is for visual verification only.
 ---
 
 # pi-browser-harness
 
-Direct browser control via CDP. Gives pi agents full control of a real Chrome browser.
-Browser tools coexist with all standard pi tools (read, bash, edit, write, etc.).
+Direct browser control of the user's running Chrome via CDP.
 
-## Setup
+## Tool hierarchy
 
-Run `/browser-setup` to install browser-harness and connect to Chrome. Chrome must be running
-with remote debugging enabled. Start Chrome with:
-
-```
---remote-debugging-port=9222
-```
-
-Check status with `/browser-status`. Restart the daemon with `/browser-reload-daemon`.
-
-## Core Workflow
-
-```
-browser_screenshot() → browser_click(x, y) → browser_screenshot()
-```
-
-1. **Screenshot first** — see what's on screen before any interaction
-2. **Coordinate click** — read the target pixel from the screenshot, click at (x, y)
-3. **Screenshot again** — verify the action worked
-
-Compositor-level clicks (`Input.dispatchMouseEvent`) work through iframes, shadow DOM,
-and cross-origin content. No selectors needed.
-
-## Parallelization
-
-**Always parallelize when possible.** The harness automatically serializes mutation tools so they never race on shared state. Emit independent tools in the same turn for better performance.
-
-### What can run in parallel
-
-- **Observation tools** (`browser_screenshot`, `browser_page_info`, `browser_execute_js`, `browser_list_tabs`, `browser_current_tab`, `browser_http_get`, `browser_print_to_pdf`) can run in parallel with each other and with mutation tools.
-
-- **Opening result pages:** Use `browser_open_urls` to open multiple URLs in parallel tabs. After opening, use `browser_list_tabs` to see the new tabs and `browser_switch_tab` to visit each one.
-
-- **API calls:** `browser_http_get` calls are independent of browser state — fire multiple GETs in parallel.
-
-- **Read + browser tools:** Use `read` and `browser_*` tools in the same turn — reading files or previous results is independent of browser actions.
-
-### What must be sequential (harness enforces this)
-
-- **Mutation tools** (`click`, `type`, `scroll`, `navigate`, `switch_tab`, `press_key`, `drag_and_drop`, `upload_file`, `go_back`, `go_forward`, `reload`, `handle_dialog`, `viewport_resize`, `wait_for_load`, `download`) are automatically queued through a mutex. Even if you emit them in parallel, they execute serially in FIFO order. You do not need to worry about race conditions.
-
-- **Tab switching then acting:** `browser_switch_tab` must complete before acting on the switched-to tab. This is enforced automatically.
-
-### Pattern: Parallel research across tabs
-
-```
-# Step 1: Open all result pages at once
-browser_open_urls(["url1", "url2", "url3"])
-
-# Step 2: Wait for them to load, then extract in parallel
-browser_list_tabs()  # get targetIds
-
-# Step 3: Extract data from each tab (parallel)
-browser_switch_tab(targetId1) → browser_screenshot()   # These are sequential per tab
-browser_switch_tab(targetId2) → browser_execute_js(...) # but you batch switch+act
-```
-
-### Pattern: Parallel observation + mutation
-
-```
-# Fire observation tools while a mutation is in progress
-browser_click(x, y)                           # mutation (queued)
-browser_screenshot()                            # observation (runs immediately in parallel)
-browser_page_info()                             # observation (runs immediately in parallel)
-browser_http_get("https://api.example.com")    # external (runs immediately in parallel)
-```
-
-## Available Tools
-
-### Navigation
-- `browser_navigate` — navigate to a URL (creates new tab on first call, reuses current tab after)
-- `browser_new_tab` — open a new tab, optionally navigate (use for first navigation)
-- `browser_open_urls` — open multiple URLs in parallel tabs (use after search to open result links)
-- `browser_go_back` / `browser_go_forward` / `browser_reload` — history navigation
-- `browser_page_info` — get URL, title, viewport, scroll position, or dialog info
-- `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` — tab management
-
-### Interaction
-- `browser_click` — click at viewport CSS-pixel coordinates
-- `browser_type` — type text into focused element
-- `browser_press_key` — press a key (Enter, Tab, Escape, arrows, etc.)
-- `browser_scroll` — scroll the page at coordinates
-
-### Visual
-- `browser_screenshot` — capture PNG or JPEG screenshot (rendered inline in the TUI)
-
-### Data Extraction
-- `browser_execute_js` — execute JavaScript and return result
-- `browser_http_get` — direct HTTP GET outside browser (10-50x faster for APIs)
-
-### Utility
-- `browser_wait` — wait N seconds
-- `browser_wait_for_load` — wait for document.readyState === 'complete'
-- `browser_handle_dialog` — accept or dismiss JS dialogs (alert/confirm/prompt)
-
-### Extending
-- `browser_run_script` — execute a temporary script file with daemon access
-
-## Pattern Reference
-
-### Navigation
-```
-browser_new_tab("https://example.com") → browser_wait_for_load() → browser_screenshot()
-```
-
-### Form Filling
-```
-browser_screenshot() → find input coordinates
-→ browser_click(x, y) → browser_type("text")
-→ browser_press_key("Tab") → browser_screenshot()
 ```
+What do you need to know?
 
-### Data Extraction
-```
-browser_execute_js("document.querySelector('.price').innerText")
-// or for APIs directly:
-browser_http_get("https://api.example.com/data")
-```
-
-### Scrolling
-```
-browser_screenshot() → browser_scroll({ deltaY: 500 }) → browser_screenshot()
-```
-Note: deltaY follows W3C wheel-event convention: positive=down, negative=up. Default deltaY=300 scrolls down.
-
-### Research (search engine + browser_open_urls)
-```
-browser_new_tab("https://google.com/search?q=query") → search engine results
-browser_open_urls({urls: ["url1", "url2"]}) → open result pages in parallel tabs
-browser_list_tabs() → see all open tabs with targetIds
-browser_switch_tab({targetId: "..."}) → switch to a specific tab
-browser_wait_for_load() → wait for the page to render
-browser_screenshot() → visually inspect the page
-browser_execute_js("document.querySelector('.content').innerText") → extract text
-```
-
-## Temporary Scripts
-
-When the built-in tools aren't enough, write a temporary script to disk and execute it
-with `browser_run_script`. Scripts run in the harness process with direct access to
-the browser daemon and full Node.js APIs — no dynamic tool registration needed.
-
-```
-  ● agent: needs to scrape 47 paginated search result pages
+  ├─ Page structure / what's clickable / labels?
+  │     → browser_snapshot     (DEFAULT — AX tree with @(x,y) per interactive element)
   │
-  ● write("/tmp/scrape-pages.js", "...script using daemon.cdp() and daemon.evaluateJS()...")
+  ├─ A specific element's value / attribute / coords?
+  │     → browser_execute_js   (e.g. el.innerText, el.getBoundingClientRect())
   │
-  ● browser_run_script("/tmp/scrape-pages.js", { urls: [...] })
+  ├─ Network behavior on the current page?
+  │     → browser_network_requests
   │
-  ✓ script executes and returns results
-```
-
-The script is written to disk, so it's auditable and re-runnable.
-
-**When to use scripts:**
-- You're about to do the same 3+ step sequence more than once
-- You need a capability the built-in tools don't provide (e.g., bulk CSV export, structured extraction, pagination)
-- You discover a domain-specific pattern that should be reusable
-- The alternative is a fragile multi-turn loop of screenshots + clicks
-
-**What NOT to do:**
-- Don't write scripts for one-off actions — use the built-in browser_* tools directly.
-- Don't try to call browser_* tools from within a script — sequence them as separate tool calls.
-
-**Script bindings:**
-- `params` — the arguments passed to browser_run_script
-- `daemon` — browser daemon with these methods:
-  - `daemon.evaluateJs(expression)` — run JS in the current page
-  - `daemon.pageInfo()` — get current page info or dialog
-  - `daemon.listTabs()` — returns `{ success, data: [{ targetId, url, title }] }`
-  - `daemon.switchTab(targetId)` — switch to a tab by full targetId
-  - `daemon.newTab(url?)` — open and switch to a new tab
-  - `daemon.current()` — returns `{ targetId, url?, title? }`
-  - `daemon.session(targetId)` — returns a session object for raw CDP:
-    - `session.call(method, params?)` — CDP command on the page target
-    - `session.callOnTarget(method, params, sessionId)` — CDP on a specific session
-    - `session.callBrowser(method, params?)` — CDP on the browser target
-    - `session.takeDialog()` — get/dismiss pending dialog
-- `require` — Node.js require() (use 'fs', 'path', 'crypto', etc.)
-- `signal` — AbortSignal for cancellation
-- `onUpdate` — progress callback: onUpdate({ content: [{ type: 'text', text: '...' }] })
-- `ctx` — ExtensionContext (ctx.cwd, ctx.signal, etc.)
-- `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`
-
-**Example — a script that extracts structured data:**
-```javascript
-// /tmp/extract-products.js
-const info = await daemon.pageInfo();
-if (info && "dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message);
-const data = await daemon.evaluateJs(`
-  JSON.stringify(Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({
-    title: el.querySelector('${params.titleSelector}')?.textContent?.trim(),
-    price: el.querySelector('${params.priceSelector}')?.textContent?.trim(),
-  })))
-`);
-return { content: [{ type: 'text', text: data }], details: { raw: JSON.parse(data) } };
-```
-
-## What NOT To Do
-
-- **Don't launch your own browser** — you're connected to the user's real Chrome
-- **Don't type credentials** — if you hit an auth wall, stop and ask the user
-- **Don't assume page state** — screenshot to confirm
-- **Don't write pixel coordinates into documentation** — describe how to locate targets (selectors, text, aria-label)
-- **Don't use browser_navigate for first navigation** — use browser_new_tab to explicitly preserve the user's active tab (browser_navigate auto-creates a new tab on first call, but explicit is clearer)
-- **Don't ignore dialogs** — if browser_page_info returns a dialog, handle it first
-
-## Dialogs
-
-JS dialogs (alert, confirm, prompt, beforeunload) freeze the page's JS thread.
-If browser_page_info returns a dialog, handle it with browser_handle_dialog
-before any other browser action.
-
-```
-// Check page_info first — if it shows a dialog:
-browser_handle_dialog({ action: "accept" })       // accept/confirm
-browser_handle_dialog({ action: "dismiss" })       // dismiss/cancel
-browser_handle_dialog({ action: "accept", promptText: "hello" })  // prompt with text
+  ├─ JS errors / why did nothing happen after an action?
+  │     → browser_console     (DIAGNOSTIC — only when something looks broken)
+  │
+  └─ Visual rendering (layout / colors / chart drew correctly)?
+        → browser_screenshot   (LAST RESORT — pixels only)
 ```
 
-## SPAs (React, Vue, etc.)
+Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. No screenshot round-trip.
 
-`browser_wait_for_load()` only checks `document.readyState === "complete"`.
-Single-page apps often paint content after this. After wait_for_load, use
-`browser_execute_js` to check for specific elements:
+## Connection
 
-```
-browser_execute_js("!!document.querySelector('.loaded-content')")
-```
+You're attached to the user's real Chrome — never launch your own. If auth is required, stop and ask the user. If `browser_page_info` returns a dialog, handle it first with `browser_handle_dialog`.
 
-## Keyboard Modifiers
+## Diagnosing a "nothing happened" moment
 
-`browser_press_key` supports modifier bitfield:
-- 1 = Alt, 2 = Ctrl, 4 = Meta/Cmd, 8 = Shift
-- Combine: Ctrl+Shift = 2|8 = 10, Cmd+Shift = 4|8 = 12
+When an action runs but the page didn't change, capture `browser_console`'s `nextCursor` *before* the action, take the action, then call `browser_console({ sinceSeq: <cursor> })` after — this isolates what your action caused from what was already there. Pair with `browser_network_requests({ sinceMs: 5000 })` to see if an API call fired and failed. The console buffer is page-scoped: it clears on tab switch, capacity 500.
 
-## Troubleshooting
+## Temporary scripts
 
-- **Daemon not starting**: Run `/browser-setup` to re-run the guided setup
-- **"DevToolsActivePort not found"**: Open chrome://inspect/#remote-debugging, tick checkbox, click Allow
-- **Stale session**: Run `/browser-reload-daemon` to restart the daemon
-- **Dialog blocking**: Run browser_page_info to detect it, then browser_handle_dialog
-- **Status check**: Run `/browser-status` to see daemon health and current page
-- **Screenshot maxDim not working**: Install sharp (`npm install sharp`) for auto-resize support
-- **Google navigation fails**: Google's anti-bot detection may reject CDP navigation; use `browser_http_get` to fetch search results instead
+When a workflow repeats 3+ times or needs Node.js APIs, write a script to disk and run it with `browser_run_script`. Scripts get a `daemon` binding for direct CDP access — much faster than chaining tool calls.
 
-## Architecture
+**Bindings inside a script:**
 
-```
-pi agent → pi-browser-harness (TypeScript)
-               │ CDP WebSocket
-               ▼
-            Chrome
-
-Temporary scripts:
-  write("/tmp/script.js") → browser_run_script("/tmp/script.js")
-      │                              │
-      ▼                              ▼
-  script on disk (auditable)    executed in harness process with daemon access
-```
+- `params` — args passed to `browser_run_script`
+- `daemon`:
+  - `daemon.evaluateJs(expression)` — run JS in the current page
+  - `daemon.pageInfo()` — `{ url, title, ... }` or `{ dialog }`
+  - `daemon.listTabs()` / `daemon.switchTab(targetId)` / `daemon.newTab(url?)` / `daemon.current()`
+  - `daemon.session(targetId)` for raw CDP: `session.call`, `session.callOnTarget`, `session.callBrowser`, `session.takeDialog`
+- `require`, `fetch`, `JSON`, `Buffer`, `console`, `setTimeout`, `clearTimeout`
+- `signal` — AbortSignal
+- `onUpdate({ content: [{ type: 'text', text }] })` — progress callback
+- `ctx` — `ExtensionContext`
+
+**Don't:**
+- Use scripts for one-off actions — call `browser_*` tools directly.
+- Call `browser_*` tools from inside a script — sequence them as separate tool calls outside.

package/src/cdp/console-buffer.ts

@@ -0,0 +1,263 @@
+/**
+ * In-memory aggregator for CDP Runtime.consoleAPICalled and Log.entryAdded events.
+ *
+ * Mirrors network-buffer.ts: the single CDP event loop in session.ts hands raw
+ * params to ingestConsoleApi / ingestLogEntry; tools later read records via drain().
+ *
+ * Bounded by capacity (default 500). On overflow the oldest record is evicted FIFO,
+ * and the overflow flag is reported once per drain so tools can warn the LLM that
+ * data may be missing.
+ *
+ * Each record carries a monotonic `seq`. Tools expose the last drained seq as
+ * `nextCursor`; callers pass it back as `sinceSeq` to read only what's new since
+ * the previous drain. This is the cursor pattern that makes "what did this action
+ * cause?" answerable in one tool call.
+ *
+ * Pure module (no CDP dependency) — joining/filtering logic stays trivially testable.
+ */
+
+export type ConsoleLevel = "log" | "info" | "warn" | "error" | "debug";
+
+export type ConsoleRecord = {
+  seq: number;
+  /** Wall-clock ms when the event was ingested, in Date.now() units. */
+  timestampMs: number;
+  level: ConsoleLevel;
+  /** Joined args (console-api) or the entry text (log-entry). Per-arg truncated. */
+  text: string;
+  source: "console-api" | "log-entry";
+  url?: string;
+  lineNumber?: number;
+  /** Top frames only, joined with newlines. Populated for error/warn when CDP provides one. */
+  stackTrace?: string;
+};
+
+export type ConsoleFilter = {
+  levels?: ConsoleLevel[];
+  /** Substring match by default. Wrap in slashes (`/foo/`) for regex. */
+  textPattern?: string;
+  /** Only records with seq strictly greater than this. */
+  sinceSeq?: number;
+  /** Only records with timestamp at most N ms ago. */
+  sinceMs?: number;
+  /** Cap on returned records (default 50, max 500). */
+  limit?: number;
+};
+
+export type ConsoleDrainResult = {
+  readonly records: ConsoleRecord[];
+  /** Total matches before `limit` was applied. */
+  readonly total: number;
+  /** True if the ring buffer evicted at least one record since the last drain. */
+  readonly bufferOverflowed: boolean;
+};
+
+export type ConsoleBuffer = {
+  ingestConsoleApi(p: unknown): void;
+  ingestLogEntry(p: unknown): void;
+  drain(filter: ConsoleFilter): ConsoleDrainResult;
+  clear(): void;
+};
+
+/** Per-arg cap to stop a single console.log(hugeBlob) from blowing the buffer. */
+const PER_ARG_CAP = 2000;
+/** Max stack frames preserved per record. */
+const MAX_STACK_FRAMES = 3;
+
+const compileTextMatcher = (pattern: string): ((text: string) => boolean) => {
+  if (pattern.length >= 2 && pattern.startsWith("/") && pattern.endsWith("/")) {
+    try {
+      const re = new RegExp(pattern.slice(1, -1));
+      return (text) => re.test(text);
+    } catch {
+      // Fall through to substring match if the pattern is not a valid regex.
+    }
+  }
+  return (text) => text.includes(pattern);
+};
+
+const truncateArg = (s: string): string =>
+  s.length > PER_ARG_CAP ? s.slice(0, PER_ARG_CAP) + "…" : s;
+
+/** Best-effort stringification of one Runtime.RemoteObject. */
+const stringifyRemoteObject = (
+  arg: { type?: string; subtype?: string; value?: unknown; description?: string; unserializableValue?: string },
+): string => {
+  if (arg.unserializableValue !== undefined) return arg.unserializableValue;
+  if (arg.value !== undefined) {
+    if (typeof arg.value === "string") return arg.value;
+    try {
+      return JSON.stringify(arg.value);
+    } catch {
+      return String(arg.value);
+    }
+  }
+  if (arg.description !== undefined) return arg.description;
+  return arg.type ?? "";
+};
+
+/** Map CDP console-API method names to our normalized levels. */
+const mapConsoleApiLevel = (type: string | undefined): ConsoleLevel => {
+  switch (type) {
+    case "error":
+    case "assert":
+      return "error";
+    case "warning":
+      return "warn";
+    case "debug":
+    case "trace":
+      return "debug";
+    case "info":
+      return "info";
+    default:
+      return "log";
+  }
+};
+
+/** Map Log.LogEntry.level values ("verbose","info","warning","error") to our levels. */
+const mapLogEntryLevel = (level: string | undefined): ConsoleLevel => {
+  switch (level) {
+    case "error":
+      return "error";
+    case "warning":
+      return "warn";
+    case "verbose":
+      return "debug";
+    default:
+      return "info";
+  }
+};
+
+const formatStackTrace = (
+  st: { callFrames?: ReadonlyArray<{ url?: string; lineNumber?: number; functionName?: string }> } | undefined,
+): string | undefined => {
+  const frames = st?.callFrames;
+  if (!frames || frames.length === 0) return undefined;
+  const top = frames.slice(0, MAX_STACK_FRAMES);
+  const rendered = top.map((f) => {
+    const fn = f.functionName !== undefined && f.functionName !== "" ? f.functionName : "<anonymous>";
+    const loc = f.url !== undefined && f.url !== "" ? `${f.url}:${(f.lineNumber ?? 0) + 1}` : "<unknown>";
+    return `  at ${fn} (${loc})`;
+  });
+  return rendered.join("\n");
+};
+
+export const createConsoleBuffer = (capacity = 500): ConsoleBuffer => {
+  // Insertion-ordered map keyed by seq. Map preserves insertion order in JS,
+  // so iteration gives us oldest-first eviction without a separate index.
+  const records = new Map<number, ConsoleRecord>();
+  let nextSeq = 1;
+  let overflowed = false;
+
+  const evictOldestIfFull = (): void => {
+    while (records.size >= capacity) {
+      const oldest = records.keys().next();
+      if (oldest.done) return;
+      records.delete(oldest.value);
+      overflowed = true;
+    }
+  };
+
+  const insert = (rec: ConsoleRecord): void => {
+    evictOldestIfFull();
+    records.set(rec.seq, rec);
+  };
+
+  return {
+    ingestConsoleApi(p) {
+      // CDP boundary cast: Runtime.consoleAPICalled
+      const params = p as
+        | {
+            type?: string;
+            args?: ReadonlyArray<{
+              type?: string;
+              subtype?: string;
+              value?: unknown;
+              description?: string;
+              unserializableValue?: string;
+            }>;
+            stackTrace?: { callFrames?: ReadonlyArray<{ url?: string; lineNumber?: number; functionName?: string }> };
+          }
+        | undefined;
+      if (!params) return;
+      const level = mapConsoleApiLevel(params.type);
+      const argTexts = (params.args ?? []).map((a) => truncateArg(stringifyRemoteObject(a)));
+      const text = argTexts.join(" ");
+      const stack = level === "error" || level === "warn" ? formatStackTrace(params.stackTrace) : undefined;
+      const top = params.stackTrace?.callFrames?.[0];
+      const rec: ConsoleRecord = {
+        seq: nextSeq++,
+        timestampMs: Date.now(),
+        level,
+        text,
+        source: "console-api",
+        ...(top?.url !== undefined && top.url !== "" ? { url: top.url } : {}),
+        ...(top?.lineNumber !== undefined ? { lineNumber: top.lineNumber + 1 } : {}),
+        ...(stack !== undefined ? { stackTrace: stack } : {}),
+      };
+      insert(rec);
+    },
+
+    ingestLogEntry(p) {
+      // CDP boundary cast: Log.entryAdded
+      const params = p as
+        | {
+            entry?: {
+              level?: string;
+              text?: string;
+              url?: string;
+              lineNumber?: number;
+              stackTrace?: { callFrames?: ReadonlyArray<{ url?: string; lineNumber?: number; functionName?: string }> };
+            };
+          }
+        | undefined;
+      const entry = params?.entry;
+      if (!entry) return;
+      const level = mapLogEntryLevel(entry.level);
+      const text = truncateArg(entry.text ?? "");
+      const stack = level === "error" || level === "warn" ? formatStackTrace(entry.stackTrace) : undefined;
+      const rec: ConsoleRecord = {
+        seq: nextSeq++,
+        timestampMs: Date.now(),
+        level,
+        text,
+        source: "log-entry",
+        ...(entry.url !== undefined && entry.url !== "" ? { url: entry.url } : {}),
+        ...(entry.lineNumber !== undefined ? { lineNumber: entry.lineNumber + 1 } : {}),
+        ...(stack !== undefined ? { stackTrace: stack } : {}),
+      };
+      insert(rec);
+    },
+
+    drain(filter) {
+      const matchText = filter.textPattern !== undefined ? compileTextMatcher(filter.textPattern) : undefined;
+      const levels = filter.levels && filter.levels.length > 0 ? new Set(filter.levels) : undefined;
+      const cutoff = filter.sinceMs !== undefined ? Date.now() - filter.sinceMs : undefined;
+      const sinceSeq = filter.sinceSeq;
+
+      const matched: ConsoleRecord[] = [];
+      for (const r of records.values()) {
+        if (sinceSeq !== undefined && r.seq <= sinceSeq) continue;
+        if (levels && !levels.has(r.level)) continue;
+        if (cutoff !== undefined && r.timestampMs < cutoff) continue;
+        if (matchText && !matchText(r.text)) continue;
+        matched.push({ ...r });
+      }
+
+      const total = matched.length;
+      const limit = Math.min(filter.limit ?? 50, 500);
+      const limited = matched.slice(-limit);
+
+      const bufferOverflowed = overflowed;
+      overflowed = false;
+
+      return { records: limited, total, bufferOverflowed };
+    },
+
+    clear() {
+      records.clear();
+      nextSeq = 1;
+      overflowed = false;
+    },
+  };
+};