browser-pilot-cli 0.1.0 → 0.1.2

package/README.md

@@ -1,89 +1,95 @@
 # browser-pilot
 
-CLI tool that gives LLM agents control of your real browser — with your logins, cookies, and extensions intact.
+Give your AI agent control of your real browser — with your logins, cookies, and extensions intact. No extension needed.
 
 ```bash
-bp open https://github.com       # navigate — returns interactive elements
-bp click 3                        # click [3] — returns updated page
-bp type 5 "hello" --submit        # type + Enter — returns updated page
-bp eval "document.title"          # run any JavaScript
+npm install -g browser-pilot-cli
 ```
 
+## Agent Setup
+
+### 1. Enable Chrome remote debugging (one-time)
+
+Open `chrome://inspect/#remote-debugging` in Chrome (144+) and click Allow. No command-line flags, no restart needed.
+
+> Chrome 136 disabled the old `--remote-debugging-port` flag for security. Chrome 144 introduced this new UI toggle as the replacement — browser-pilot uses this.
+
+### 2. Install the plugin for your agent
+
+**Claude Code:**
+```
+/plugin marketplace add https://github.com/relixiaobo/browser-pilot.git
+/plugin install browser-pilot@browser-pilot-marketplace
+```
+
+**Codex CLI:**
+```bash
+npx skills add relixiaobo/browser-pilot
+```
+
+**OpenClaw:**
+```bash
+cp -r plugin/skills/browser-pilot ~/.agents/skills/
+```
+
+**Cursor / VS Code Copilot:**
+```bash
+npx skills add relixiaobo/browser-pilot
+```
+
+### 3. Use it
+
+Just tell your agent what you want to do:
+
+- "Open GitHub and check my notifications"
+- "Go to Hacker News and summarize the top 5 posts"
+- "Fill out the form on this page"
+
+The agent will use `bp` commands automatically. Your real login sessions are preserved — no need to re-authenticate.
+
+## Why browser-pilot?
+
+- **No extension required** — Uses Chrome 144's native remote debugging toggle, not the Extension Debugger API
+- **Real login sessions** — Operates your actual browser profile. Cookies, extensions, logins all intact
+- **CLI-native** — Any agent with bash access can use it. No MCP protocol, no SDK integration needed
+- **Auto-snapshot** — Every action returns page state with numbered `[ref]` elements, so the agent always knows what's on screen
+- **Lightweight** — 78KB npm package. No bundled Chromium (unlike Playwright's 400MB+)
+- **Rich editor support** — Works with contenteditable editors (Draft.js, ProseMirror, Quill, Slate) and Shadow DOM elements out of the box
+
 ## Comparison
 
 | | browser-pilot | Playwright MCP | Chrome DevTools MCP | browser-use |
 |---|---|---|---|---|
 | **Interface** | CLI (bash) | MCP protocol | MCP protocol | Python SDK |
 | **Login session reuse** | Yes | No | Depends | Yes |
+| **Extension required** | No | No | No | No |
 | **Element refs** | Numbered (accessibility tree) | Named refs (ARIA) | CSS selectors | Numbered (DOM) |
 | **Auto-snapshot after action** | Yes | Yes | No | Yes |
 | **Network interception** | Yes (block/mock/headers) | Yes | Yes | No |
-| **Drag and drop** | No | Yes | Yes | No |
-| **Mobile emulation** | No | Yes | Yes | Yes |
 | **Multi-browser** | Chromium-only | Chromium + Firefox + WebKit | Chromium-only | Chromium-only |
-| **Connection model** | Daemon (one-time Allow) | MCP server | MCP server | Python process |
 | **Dialog auto-handling** | Yes | Yes | No | Yes |
 | **JSON output** | Default | MCP structured | MCP structured | Python objects |
-| **Visual indicator** | Pulsing glow | None | None | None |
 | **File upload** | Auto-detect input | Yes | No | Yes |
-| **Autonomous agent** | No (tool, not agent) | No (tool) | No (tool) | Yes (plans + executes) |
-
-**browser-pilot is best when:**
-- You need your existing login sessions (paywalled content, internal tools)
-- Your LLM has bash access but no MCP support
-- You want every action to return page state automatically
-
-**Use Playwright MCP when:**
-- You need multi-browser or mobile emulation
-- Your LLM supports MCP natively
-- You don't need existing login sessions
-
-**Use browser-use when:**
-- You want an autonomous agent that plans and executes multi-step tasks
-- You're building in Python
-
-## Quick Start
-
-```bash
-# Install
-npm install -g browser-pilot
-
-# Enable debugging in Chrome (one-time)
-# Open chrome://inspect/#remote-debugging → toggle ON
-
-# Connect (click Allow in Chrome's dialog)
-bp connect
-
-# Use
-bp open https://github.com
-bp click 3
-bp type 5 "hello" --submit
-bp eval "document.title"
-bp screenshot page.png
-bp disconnect
-```
 
 ## How It Works
 
 ```
-LLM (bash tool)
+Agent (bash tool)
   │  bp open / bp click / bp eval ...
   ▼
 CLI Process ──── HTTP/Unix Socket ──── Daemon Process (persistent)
                                            │
-                                           │  WebSocket (CDP, one-time Allow)
+                                           │  WebSocket (CDP)
                                            ▼
                                        Chrome (your browser, your profile)
                                        ├── Your windows (untouched)
-                                       └── Pilot window (bp operates here)
+                                       └── Pilot window (agent operates here)
 ```
 
-The daemon maintains a single CDP WebSocket connection. Chrome's "Allow" dialog appears once per session. A pulsing blue glow around the Pilot window indicates the agent is active.
+The daemon maintains a single CDP WebSocket connection. A pulsing blue glow around the Pilot window indicates the agent is active.
 
 ## Commands
 
-Run `bp --help` for full details including workflow, refs, and eval examples.
-
 ### Core Loop
 
 | Command | Returns | Description |
@@ -122,7 +128,7 @@ Popup windows (target="_blank", window.open) are auto-detected. Run `bp tabs` to
 | `bp net` | List recent requests (`--url`, `--method`, `--status`, `--type`) |
 | `bp net show <id>` | Full request/response details (`--save <file>`) |
 | `bp net block <pattern>` | Block requests matching URL pattern |
-| `bp net mock <pattern>` | Mock responses (`--body`, `--file`, `--status`) |
+| `bp net mock <pattern>` | Mock responses (`--body`, `--file`) |
 | `bp net headers <pattern> <header...>` | Add/override request headers |
 | `bp net rules` | List active interception rules |
 | `bp net remove [id]` | Remove rule(s) (`--all`) |
@@ -144,13 +150,18 @@ Action commands return a snapshot of interactive elements, each with a `[ref]` n
 
 ```
 [1] link "Home"
-[2] textbox "Search"
-[3] button "Submit"
+[2] textbox "Search"                ← <input>, <textarea>, or contenteditable
+[3] textbox ""                      ← unnamed input (still interactive)
+[4] combobox ""                     ← <select> dropdown
+[5] spinbutton "Quantity"           ← <input type="number">
+[6] button "Submit"
+[7] checkbox "Agree" checked
+[8] slider "Volume"                 ← <input type="range">
 ```
 
 Use the number in subsequent commands: `bp click 1`, `bp type 2 "hello"`.
 
-Refs are scoped to the current page — they refresh automatically after every action.
+Refs are scoped to the current page — they refresh automatically after every action. Elements inside Shadow DOM are included automatically.
 
 ## Output
 
@@ -192,6 +203,22 @@ bp click 5                        # click "Search by image"
 bp upload ~/Downloads/photo.jpg    # auto-finds file input, triggers upload
 ```
 
+## Rich Text Editors & Shadow DOM
+
+`bp type` works with contenteditable-based editors (Draft.js, ProseMirror, Quill, Slate, Lexical). They appear as `textbox` in snapshots:
+
+```bash
+bp type 3 "new content" --clear     # replace content in a rich text editor
+```
+
+Shadow DOM elements are traversed automatically — no special commands needed. Elements inside open shadow roots (even deeply nested) appear in snapshots and can be clicked/typed normally.
+
+For `<select>` dropdowns (shown as `combobox`), use `bp eval`:
+
+```bash
+bp eval 'document.querySelector("select").value = "opt2"; document.querySelector("select").dispatchEvent(new Event("change",{bubbles:true}))'
+```
+
 ## Network Interception
 
 Monitor, block, and mock HTTP requests:
@@ -208,7 +235,7 @@ bp net block "*ads*"
 
 # Mock API responses
 bp net mock "*api/data*" --body '{"ok":true}'
-bp net mock "*api/users*" --file mock.json --status 200
+bp net mock "*api/users*" --file mock.json
 
 # Override request headers
 bp net headers "*api*" "Authorization:Bearer test123"
@@ -220,11 +247,28 @@ bp net remove --all                    # clear all rules
 bp net clear                           # clear captured request log
 ```
 
+## Testing
+
+121 tests across 4 Playwright projects, adapted from Playwright/Puppeteer test fixtures and real-world sites:
+
+```bash
+npm test                    # core + compat + network (105 tests, ~2min)
+npm run test:integration    # real-site tests against the-internet.herokuapp.com
+npm run test:all            # all 121 tests
+```
+
+| Project | Tests | Coverage |
+|---------|-------|----------|
+| **core** | 41 | lifecycle, nav, click, type, press, eval, screenshot, pdf, cookies, frames, upload, auth, tabs, dialogs |
+| **compat** | 46 | contenteditable (5 variants), Shadow DOM (3 levels), input types, scrollable, overlay, select |
+| **network** | 19 | request monitoring, block, mock, headers, rule management |
+| **integration** | 15 | checkboxes, dropdown, key presses, dynamic controls, Shadow DOM, nested frames, TinyMCE, large DOM |
+
 ## Requirements
 
-- Chrome / Chromium / Edge / Brave (any Chromium-based browser)
+- Chrome 144+ / Edge / Brave (any Chromium-based browser)
 - Node.js >= 18
-- Chrome remote debugging enabled (`chrome://inspect/#remote-debugging`)
+- Remote debugging enabled (`chrome://inspect/#remote-debugging`)
 
 ## License
 

package/dist/cli.js

@@ -245,6 +245,14 @@ var SET_VALUE = `function(text, clear) {
   this.dispatchEvent(new Event('input',  {bubbles:true}));
   this.dispatchEvent(new Event('change', {bubbles:true}));
 }`;
+var IS_CONTENTEDITABLE = `function() {
+  return this.isContentEditable && !(this instanceof HTMLInputElement) && !(this instanceof HTMLTextAreaElement);
+}`;
+var CONTENTEDITABLE_CLEAR = `function() {
+  this.focus();
+  document.execCommand('selectAll');
+  document.execCommand('delete');
+}`;
 var PAGE_INFO = `JSON.stringify({title:document.title,url:location.href})`;
 var PAGE_DIMENSIONS = `JSON.stringify({
   width:  Math.max(document.documentElement.scrollWidth,  document.documentElement.clientWidth),
@@ -431,6 +439,17 @@ var INTERACTIVE_ROLES = /* @__PURE__ */ new Set([
   "menuitemradio",
   "tab"
 ]);
+var ALLOW_EMPTY_NAME = /* @__PURE__ */ new Set([
+  "textbox",
+  "searchbox",
+  "combobox",
+  "listbox",
+  "checkbox",
+  "radio",
+  "spinbutton",
+  "slider",
+  "switch"
+]);
 function saveRefs(targetId, entries) {
   writeFileSync2(REFS_FILE, JSON.stringify({ targetId, entries }), { mode: 384 });
 }
@@ -466,16 +485,20 @@ async function takeSnapshot(transport, sessionId, targetId, limit = 50) {
     if (!node) return;
     if (!node.ignored) {
       const role = node.role?.value;
-      if (role && INTERACTIVE_ROLES.has(role) && node.backendDOMNodeId !== void 0) {
-        const props = Object.fromEntries(
-          (node.properties || []).map((p) => [p.name, p.value?.value])
-        );
+      const props = Object.fromEntries(
+        (node.properties || []).map((p) => [p.name, p.value?.value])
+      );
+      const isEditable = props.editable === "richtext";
+      const isInteractive = role && (INTERACTIVE_ROLES.has(role) || isEditable);
+      if (isInteractive && node.backendDOMNodeId !== void 0) {
         const name = node.name?.value || "";
         const value = node.value?.value;
-        if (!props.disabled && (name || value) && refs.length < limit) {
+        const effectiveRole = isEditable && role === "generic" ? "textbox" : role;
+        const hasIdentity = name || value || ALLOW_EMPTY_NAME.has(effectiveRole) || isEditable;
+        if (!props.disabled && hasIdentity && refs.length < limit) {
           const checked = props.checked === "true" || props.checked === true ? true : void 0;
-          refs.push({ backendNodeId: node.backendDOMNodeId, role, name });
-          elements.push({ ref: refs.length, backendNodeId: node.backendDOMNodeId, role, name, value, checked });
+          refs.push({ backendNodeId: node.backendDOMNodeId, role: effectiveRole, name });
+          elements.push({ ref: refs.length, backendNodeId: node.backendDOMNodeId, role: effectiveRole, name, value, checked });
         }
       }
     }
@@ -747,11 +770,36 @@ program.command("type <ref> <text>").description("Type text into element and ret
   await withPilot(async ({ transport, sessionId, state }) => {
     const objectId = await resolveTarget(transport, sessionId, ref, state.activeTargetId);
     try {
-      await transport.send("Runtime.callFunctionOn", {
+      const { result: ceResult } = await transport.send("Runtime.callFunctionOn", {
         objectId,
-        functionDeclaration: SET_VALUE,
-        arguments: [{ value: text }, { value: !!opts.clear }]
+        functionDeclaration: IS_CONTENTEDITABLE,
+        returnByValue: true
       }, sessionId);
+      if (ceResult.value) {
+        if (opts.clear) {
+          await transport.send("Runtime.callFunctionOn", {
+            objectId,
+            functionDeclaration: CONTENTEDITABLE_CLEAR
+          }, sessionId);
+        } else {
+          await transport.send("Runtime.callFunctionOn", {
+            objectId,
+            functionDeclaration: `function() {
+                this.focus();
+                const sel = window.getSelection();
+                sel.selectAllChildren(this);
+                sel.collapseToEnd();
+              }`
+          }, sessionId);
+        }
+        await transport.send("Input.insertText", { text }, sessionId);
+      } else {
+        await transport.send("Runtime.callFunctionOn", {
+          objectId,
+          functionDeclaration: SET_VALUE,
+          arguments: [{ value: text }, { value: !!opts.clear }]
+        }, sessionId);
+      }
       if (opts.submit) await dispatchKey(transport, sessionId, "Enter");
     } finally {
       await transport.send("Runtime.releaseObject", { objectId }, sessionId).catch(() => {
@@ -760,6 +808,73 @@ program.command("type <ref> <text>").description("Type text into element and ret
     emitSnapshot(await snap(transport, sessionId, state.activeTargetId, limit));
   });
 }));
+async function typeViaKeyboard(t, sid, text) {
+  for (const char of text) {
+    if (char === "\n") {
+      await dispatchKey(t, sid, "Enter");
+    } else if (char === "	") {
+      await dispatchKey(t, sid, "Tab");
+    } else if (char.charCodeAt(0) >= 32 && char.charCodeAt(0) <= 126) {
+      const code = char === " " ? "Space" : `Key${char.toUpperCase()}`;
+      const keyCode = char.toUpperCase().charCodeAt(0);
+      await t.send("Input.dispatchKeyEvent", { type: "keyDown", key: char, code, windowsVirtualKeyCode: keyCode, text: char }, sid);
+      await t.send("Input.dispatchKeyEvent", { type: "keyUp", key: char, code, windowsVirtualKeyCode: keyCode }, sid);
+    } else {
+      await t.send("Input.insertText", { text: char }, sid);
+    }
+  }
+}
+program.command("keyboard <text>").description("Type text via keyboard events (for canvas editors like Google Docs)").option("-c, --clear", "select all + delete before typing").option("-s, --submit", "press Enter after typing").option("-d, --delay <ms>", "delay between keystrokes in ms").option("--click <selector>", "click element by CSS selector first to focus it").option("-l, --limit <n>", "max elements in snapshot", "50").addHelpText("after", `
+Unlike 'type', this does not target a specific element. It sends real
+keyboard events to whatever is currently focused \u2014 works with canvas-based
+editors (Google Docs, Google Sheets, Figma) that don't expose DOM inputs.
+
+Use --click to focus an element before typing (sends a real CDP mouse click).
+
+Examples:
+  bp keyboard "hello world"
+  bp keyboard "new content" --clear
+  bp keyboard "search query" --submit
+  bp keyboard "Hello Docs!" --click ".kix-appview-editor"
+  bp keyboard "slow typing" --delay 50`).action(action(async (text, opts) => {
+  const limit = parseLimit(opts.limit);
+  await withPilot(async ({ transport, sessionId, state }) => {
+    if (opts.click) {
+      const { result } = await transport.send("Runtime.evaluate", {
+        expression: `JSON.stringify((function(){var el=document.querySelector(${JSON.stringify(opts.click)});if(!el)return null;el.scrollIntoView({block:'center'});var r=el.getBoundingClientRect();return{x:r.x+r.width/2,y:r.y+r.height/2}})())`,
+        returnByValue: true
+      }, sessionId);
+      const coords = result.value ? JSON.parse(result.value) : null;
+      if (!coords) throw new Error(`Element not found: ${opts.click}`);
+      await dispatchClick(transport, sessionId, coords.x, coords.y);
+      await new Promise((r) => setTimeout(r, 300));
+    }
+    if (opts.clear) {
+      await dispatchKey(transport, sessionId, "Meta+a");
+      await dispatchKey(transport, sessionId, "Delete");
+    }
+    if (opts.delay) {
+      const delay = parseInt(opts.delay, 10);
+      for (const char of text) {
+        if (char === "\n") {
+          await dispatchKey(transport, sessionId, "Enter");
+        } else if (char.charCodeAt(0) >= 32 && char.charCodeAt(0) <= 126) {
+          const code = char === " " ? "Space" : `Key${char.toUpperCase()}`;
+          const keyCode = char.toUpperCase().charCodeAt(0);
+          await transport.send("Input.dispatchKeyEvent", { type: "keyDown", key: char, code, windowsVirtualKeyCode: keyCode, text: char }, sessionId);
+          await transport.send("Input.dispatchKeyEvent", { type: "keyUp", key: char, code, windowsVirtualKeyCode: keyCode }, sessionId);
+        } else {
+          await transport.send("Input.insertText", { text: char }, sessionId);
+        }
+        await new Promise((r) => setTimeout(r, delay));
+      }
+    } else {
+      await typeViaKeyboard(transport, sessionId, text);
+    }
+    if (opts.submit) await dispatchKey(transport, sessionId, "Enter");
+    emitSnapshot(await snap(transport, sessionId, state.activeTargetId, limit));
+  });
+}));
 program.command("press <key>").description("Press key combo (e.g. Enter, Escape, Control+a) and return snapshot").option("-l, --limit <n>", "max elements in snapshot", "50").addHelpText("after", "\nKeys: Enter, Escape, Tab, Space, Backspace, Delete,\n      ArrowUp, ArrowDown, ArrowLeft, ArrowRight,\n      Home, End, PageUp, PageDown\nModifiers: Control (Ctrl), Shift, Alt, Meta (Cmd)\n\nExamples:\n  bp press Enter\n  bp press Escape\n  bp press Control+a\n  bp press Meta+c").action(action(async (key, opts) => {
   const limit = parseLimit(opts.limit);
   await withPilot(async ({ transport, sessionId, state }) => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "browser-pilot-cli",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "CLI tool to control your browser via Chrome DevTools Protocol",
   "type": "module",
   "bin": {
@@ -11,13 +11,16 @@
     "build": "tsup",
     "dev": "tsup --watch",
     "start": "node dist/cli.js",
-    "test": "bash tests/run.sh"
+    "test": "tsup && npx playwright test --project core --project compat --project network",
+    "test:integration": "tsup && npx playwright test --project integration",
+    "test:all": "tsup && npx playwright test"
   },
   "dependencies": {
     "commander": "^13.1.0",
     "ws": "^8.18.0"
   },
   "devDependencies": {
+    "@playwright/test": "^1.59.1",
     "@types/node": "^22.0.0",
     "@types/ws": "^8.5.0",
     "tsup": "^8.0.0",