btcp-browser-agent 0.1.0

package/CLAUDE.md

@@ -0,0 +1,230 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+BTCP Browser Agent is a browser-native implementation for AI agents to interact with web pages. It provides a clean separation between browser-level operations (tabs, navigation, screenshots) and DOM-level operations (clicking, typing, reading elements).
+
+## Build Commands
+
+```bash
+# Build all packages
+npm run build
+
+# Build individual packages (in order)
+npm run build:packages
+
+# Build Chrome extension example
+cd examples/chrome-extension && npm run build
+
+# Watch mode for extension development
+cd examples/chrome-extension && npm run watch
+
+# Clean build artifacts
+npm run clean
+```
+
+## Testing
+
+```bash
+# Run all tests
+npm test
+
+# Watch mode for test development
+npm run test:watch
+
+# Type checking only
+npm run typecheck
+```
+
+## Architecture
+
+### Three-Layer Command Flow
+
+Commands flow through three distinct layers:
+
+1. **Client Layer** (`packages/extension/src/index.ts`)
+   - `createClient()` provides high-level API methods
+   - Sends commands via `chrome.runtime.sendMessage`
+   - Used in popups, external scripts
+
+2. **Background Layer** (`packages/extension/src/background.ts`)
+   - `BackgroundAgent` handles browser-level operations (tabs, navigation, screenshots)
+   - Routes DOM commands to ContentAgent via `chrome.tabs.sendMessage`
+   - `SessionManager` maintains tab group state across extension lifecycle
+
+3. **Content Layer** (`packages/core/src/actions.ts`)
+   - `DOMActions` executes all DOM operations
+   - `createSnapshot()` generates accessibility tree with element refs
+   - Runs in content script context (one per tab)
+
+### Command Type System
+
+Commands are strongly typed with a discriminated union:
+
+- **CoreAction** (46 actions in `packages/core/src/types.ts`): DOM operations like `click`, `type`, `snapshot`, `highlight`
+- **ExtensionAction** (11 actions in `packages/extension/src/types.ts`): Browser operations like `navigate`, `screenshot`, `tabNew`
+- All commands extend `BaseCommand` with `id` and `action` fields
+
+### Element References and Snapshot API
+
+The snapshot system creates stable element references:
+
+```typescript
+// Snapshot returns a string (accessibility tree with embedded @ref:N markers)
+const tree = await client.snapshot();
+// Example output: "- BUTTON \"Submit\" [@ref:1]"
+
+// Use refs in commands: { action: 'click', selector: '@ref:5' }
+```
+
+**Important**: The snapshot API returns a `string` directly (the accessibility tree). Element refs are embedded as `@ref:N` markers in the tree and stored internally in a `RefMap` (WeakRef-based for memory management). Refs persist across commands within the same content script session and are used internally for the `highlight` feature.
+
+## Package Structure
+
+```
+btcp-browser-agent/           # Monorepo root
+├── packages/
+│   ├── core/                 # @btcp/core - DOM operations (ContentAgent)
+│   │   ├── src/actions.ts    # DOMActions class - all DOM command handlers
+│   │   ├── src/snapshot.ts   # Accessibility tree generation
+│   │   ├── src/types.ts      # CoreAction types (46 actions)
+│   │   └── src/ref-map.ts    # Element reference management
+│   │
+│   ├── extension/            # @btcp/extension - Browser operations
+│   │   ├── src/background.ts        # BackgroundAgent (service worker)
+│   │   ├── src/session-manager.ts   # Tab group persistence
+│   │   ├── src/content.ts           # Content script setup
+│   │   └── src/index.ts             # Client API + types
+│   │
+│   └── cli/                  # @btcp/cli - Command-line interface
+│       ├── src/commands/     # 28 CLI command implementations
+│       ├── src/parser.ts     # Natural language command parser
+│       └── src/executor.ts   # Command execution + suggestions
+│
+└── examples/
+    ├── chrome-extension/     # Full extension demo (popup + background + content)
+    └── snapshots/           # Snapshot generation utilities
+```
+
+## Key Concepts
+
+### Highlight vs CLI Commands
+
+The core `DOMActions` class supports `highlight` and `clearHighlight` actions, but these are NOT registered as CLI commands. To use them:
+
+- In popup: Use `client.execute({ id: '...', action: 'highlight' })`
+- NOT: `cli.execute('highlight')` (will fail with "Unknown command")
+
+The CLI package (`packages/cli/src/commands/index.ts`) has 28 registered commands. If a core action is missing from the CLI registry, use the raw `client.execute()` method.
+
+### Session Management
+
+The extension uses Chrome Tab Groups for session isolation:
+
+- `SessionManager` persists active session to `chrome.storage.session`
+- On extension reload, reconnects to stored group if it still exists
+- All extension operations are scoped to the active session's tabs
+- Create session: `client.groupCreate()` → returns `{ group: GroupInfo }`
+
+### Multi-Tab Operations
+
+Background agent supports two patterns for multi-tab work:
+
+```typescript
+// Pattern 1: tab() handle - interact without switching active tab
+const tab = agent.tab(tabId);
+await tab.snapshot();
+await tab.click('@ref:5');
+
+// Pattern 2: Pass tabId in execute options
+await agent.execute(command, { tabId });
+```
+
+Both avoid unnecessary tab switching for better performance.
+
+### Snapshot Formats
+
+Snapshots support two output formats:
+
+- `format: 'tree'` (default): Flat accessibility tree with refs
+- `format: 'html'`: HTML structure with semantic xpaths
+
+The tree format is optimized for AI consumption and includes role-based element descriptions.
+
+## Development Workflow
+
+### Working on Core Package
+
+```bash
+# 1. Make changes in packages/core/src/
+# 2. Build core package
+npm run build:packages
+
+# 3. Run tests
+npm test -- packages/core
+
+# 4. If working on extension example, rebuild it
+cd examples/chrome-extension && npm run build
+```
+
+### Working on Extension Example
+
+```bash
+cd examples/chrome-extension
+
+# Watch mode - auto-rebuilds on changes
+npm run watch
+
+# Load unpacked extension in Chrome:
+# 1. Open chrome://extensions
+# 2. Enable "Developer mode"
+# 3. Click "Load unpacked"
+# 4. Select examples/chrome-extension/dist/
+```
+
+### Adding New Core Actions
+
+1. Add action type to `CoreAction` in `packages/core/src/types.ts`
+2. Create command interface extending `BaseCommand`
+3. Add case to `DOMActions.dispatch()` in `packages/core/src/actions.ts`
+4. Implement handler method in `DOMActions` class
+5. Optionally create CLI command wrapper in `packages/cli/src/commands/`
+
+### Adding New CLI Commands
+
+1. Create command file in `packages/cli/src/commands/[name].ts`
+2. Export `CommandHandler` with `name`, `description`, `execute()`, `examples`
+3. Register in `packages/cli/src/commands/index.ts`
+4. Add suggestions mapping in `packages/cli/src/suggestions.ts` if needed
+
+## Common Pitfalls
+
+1. **CLI vs Client API**: CLI commands are a subset of core actions. Use `client.execute()` for actions not in CLI registry.
+
+2. **Ref Lifecycle**: Element refs are cleared on navigation and only valid within the content script that created them.
+
+3. **Cross-Origin**: ContentAgent only works same-origin when used standalone. Extension context bypasses this via content scripts.
+
+4. **Build Order**: Always build packages before examples (`npm run build:packages` then `cd examples/chrome-extension && npm run build`).
+
+5. **Session Persistence**: Session state survives extension reload via `chrome.storage.session`, but actual tab groups can be deleted by the user.
+
+## Import Paths
+
+The package exports multiple entry points:
+
+```typescript
+// Main re-exports (convenience)
+import { createContentAgent } from '@btcp/browser-agent';
+
+// Direct package imports (preferred for clarity)
+import { createContentAgent } from '@btcp/browser-agent/core';
+import { BackgroundAgent, createClient } from '@btcp/browser-agent/extension';
+import { createCLI } from '@btcp/browser-agent/cli';
+
+// Extension-specific entry points
+import '@btcp/browser-agent/extension/content';    // Content script setup
+import '@btcp/browser-agent/extension/background'; // Background script setup
+```

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 browser-tool-calling-protocol
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,309 @@
+# btcp-browser-agent
+
+Give AI agents the power to see and control any browser.
+
+A lightweight foundation for building AI systems that need browser access — automation, testing, web agents, or any browser-based workflow.
+
+## Why This Package?
+
+AI agents struggle with browsers because:
+- Raw HTML is too noisy (thousands of nodes)
+- CSS selectors break when layouts change
+- No stable way to reference elements across turns
+
+**Browser Agent solves this with smart snapshots:**
+
+```
+BUTTON "Submit" [@ref:0]
+TEXTBOX "Email" [required] [@ref:1]
+LINK "Forgot password?" [@ref:2]
+```
+
+One command gives your agent a clean, semantic view of any page. Stable `@ref` markers let it interact without fragile selectors.
+
+## Features
+
+- **Smart Snapshots** - Accessibility tree format optimized for AI comprehension
+- **Stable Element Refs** - `@ref:N` markers that survive DOM changes within a session
+- **Full Browser Control** - Navigation, tabs, screenshots, keyboard/mouse
+- **46 DOM Actions** - Click, type, fill, scroll, hover, and more
+- **Two Modes** - Chrome extension (full control) or standalone (same-origin)
+
+## Quick Example
+
+```typescript
+import { createClient } from 'btcp-browser-agent/extension';
+
+const agent = createClient();
+
+// Navigate and understand the page
+await agent.navigate('https://example.com');
+const snapshot = await agent.snapshot();
+// Returns: BUTTON "Login" [@ref:0], TEXTBOX "Email" [@ref:1], ...
+
+// Interact using refs - no CSS selectors needed
+await agent.fill('@ref:1', 'user@example.com');
+await agent.click('@ref:0');
+```
+
+## Use Cases
+
+- **AI Assistants** - Let LLMs browse the web and complete tasks for users
+- **Browser Agents** - Foundation for autonomous web agents that research, navigate, and act
+- **Automated Testing** - Reliable UI tests with stable element refs that don't break on layout changes
+- **Web Automation** - Form filling, data extraction, multi-step workflow automation
+- **Web Scraping** - Extract structured data with semantic understanding of page content
+
+## Installation
+
+```bash
+npm install btcp-browser-agent
+```
+
+## Usage Modes
+
+### Extension Mode (Full Browser Control)
+
+For Chrome extensions with cross-origin access, tab management, and screenshots.
+
+**Background Script:**
+```typescript
+import { BackgroundAgent, setupMessageListener } from 'btcp-browser-agent/extension';
+
+// Option 1: Just set up message routing
+setupMessageListener();
+
+// Option 2: Use BackgroundAgent directly for programmatic control
+const agent = new BackgroundAgent();
+await agent.navigate('https://example.com');
+await agent.screenshot();
+```
+
+**Content Script:**
+```typescript
+import { createContentAgent } from 'btcp-browser-agent';
+
+const agent = createContentAgent();
+
+// Take a snapshot
+const { data } = await agent.execute({ id: '1', action: 'snapshot' });
+console.log(data.tree);  // Accessibility tree with refs
+
+// Click an element using ref from snapshot
+await agent.execute({ id: '2', action: 'click', selector: '@ref:5' });
+```
+
+**Popup (sending commands via messaging):**
+```typescript
+import { createClient } from 'btcp-browser-agent';
+
+const client = createClient();
+
+// Navigate and interact
+await client.navigate('https://example.com');
+const snapshot = await client.snapshot();
+await client.click('@ref:5');
+const screenshot = await client.screenshot();
+```
+
+### Standalone Mode (No Extension)
+
+For use directly in a web page (limited to same-origin, no tab management):
+
+```typescript
+import { createContentAgent } from 'btcp-browser-agent';
+
+const agent = createContentAgent();
+
+// Take a snapshot
+const { data } = await agent.execute({ id: '1', action: 'snapshot' });
+
+// Interact with elements
+await agent.execute({ id: '2', action: 'click', selector: '@ref:5' });
+await agent.execute({ id: '3', action: 'fill', selector: '@ref:3', value: 'Hello' });
+```
+
+## API Reference
+
+### BackgroundAgent (Extension Background Script)
+
+High-level browser orchestrator that runs in the extension's background script.
+
+```typescript
+import { BackgroundAgent } from 'btcp-browser-agent/extension';
+
+const agent = new BackgroundAgent();
+
+// Tab Management
+await agent.newTab({ url: 'https://example.com' });
+await agent.switchTab(tabId);
+await agent.closeTab(tabId);
+const tabs = await agent.listTabs();
+
+// Navigation
+await agent.navigate('https://example.com');
+await agent.back();
+await agent.forward();
+await agent.reload();
+
+// Screenshots
+const screenshot = await agent.screenshot({ format: 'png' });
+
+// Execute commands (routes to ContentAgent for DOM operations)
+await agent.execute({ id: '1', action: 'click', selector: '#submit' });
+```
+
+#### Multi-Tab Operations
+
+```typescript
+// Open tabs
+const tab1 = await agent.newTab({ url: 'https://google.com' });
+const tab2 = await agent.newTab({ url: 'https://github.com', active: false });
+
+// Method 1: tab() handle - interact without switching
+const githubTab = agent.tab(tab2.id);
+await githubTab.snapshot();
+await githubTab.click('@ref:5');
+
+// Method 2: Specify tabId in execute
+await agent.execute(
+  { id: '1', action: 'getText', selector: 'h1' },
+  { tabId: tab2.id }
+);
+
+// Active tab stays tab1 (no switching needed)
+```
+
+### ContentAgent (Content Script)
+
+DOM automation agent that runs in content scripts or web pages.
+
+```typescript
+import { createContentAgent } from 'btcp-browser-agent';
+
+const agent = createContentAgent();
+
+// Execute commands
+const response = await agent.execute({
+  id: 'cmd1',
+  action: 'snapshot'
+});
+```
+
+#### Available Actions
+
+**DOM Reading:**
+| Action | Description |
+|--------|-------------|
+| `snapshot` | Get accessibility tree with element refs |
+| `getText` | Get element text content |
+| `getAttribute` | Get element attribute value |
+| `isVisible` | Check if element is visible |
+| `isEnabled` | Check if element is enabled |
+| `isChecked` | Check if checkbox/radio is checked |
+| `getBoundingBox` | Get element dimensions |
+
+**Element Interaction:**
+| Action | Description |
+|--------|-------------|
+| `click` | Click an element |
+| `dblclick` | Double-click an element |
+| `type` | Type text (keystroke by keystroke) |
+| `fill` | Fill input (instant) |
+| `clear` | Clear input value |
+| `check` | Check checkbox |
+| `uncheck` | Uncheck checkbox |
+| `select` | Select dropdown option |
+| `hover` | Hover over element |
+| `focus` | Focus element |
+| `blur` | Remove focus |
+
+**Keyboard/Mouse:**
+| Action | Description |
+|--------|-------------|
+| `press` | Press a key |
+| `keyDown` | Key down event |
+| `keyUp` | Key up event |
+
+**Other:**
+| Action | Description |
+|--------|-------------|
+| `scroll` | Scroll page or element |
+| `scrollIntoView` | Scroll element into view |
+| `wait` | Wait for element state |
+| `evaluate` | Execute JavaScript |
+
+### Element Refs
+
+The `snapshot` action returns element references for stable selection:
+
+```typescript
+const { data } = await agent.execute({ id: '1', action: 'snapshot' });
+// data.tree: "BUTTON 'Submit' [@ref:5]\nTEXTBOX 'Email' [@ref:3]"
+
+// Use refs in subsequent commands
+await agent.execute({ id: '2', action: 'click', selector: '@ref:5' });
+```
+
+## Architecture
+
+The package provides a clean separation between browser-level and DOM-level operations:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  Background Script (Extension Service Worker)                    │
+│  ┌─────────────────────────────────────────────────────────────┐│
+│  │ BackgroundAgent                                              ││
+│  │  - Tab management (create, close, switch, list)             ││
+│  │  - Navigation (goto, back, forward, reload)                 ││
+│  │  - Screenshots (chrome.tabs.captureVisibleTab)              ││
+│  │  - Routes DOM commands → ContentAgent                       ││
+│  └─────────────────────────────────────────────────────────────┘│
+└─────────────────────────────────────────────────────────────────┘
+                              │
+            chrome.tabs.sendMessage
+                              ▼
+┌─────────────────────────────────────────────────────────────────┐
+│  Content Script (Per Tab)                                        │
+│  ┌─────────────────────────────────────────────────────────────┐│
+│  │ ContentAgent                                                 ││
+│  │  - DOM snapshot (accessibility tree)                        ││
+│  │  - Element interaction (click, type, fill, hover)           ││
+│  │  - DOM queries (getText, getAttribute, isVisible)           ││
+│  │  - Keyboard/mouse events                                    ││
+│  └─────────────────────────────────────────────────────────────┘│
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Package Structure
+
+```
+btcp-browser-agent/
+├── @btcp/core          # ContentAgent - DOM operations
+│   ├── createContentAgent()
+│   ├── DOMActions
+│   └── createSnapshot()
+│
+├── @btcp/extension     # BackgroundAgent - Browser operations
+│   ├── BackgroundAgent
+│   ├── setupMessageListener()
+│   └── createClient()
+│
+└── btcp-browser-agent   # Main package - re-exports both
+```
+
+## Capabilities Comparison
+
+| Capability | ContentAgent (Standalone) | BackgroundAgent (Extension) |
+|------------|--------------------------|--------------------------|
+| DOM Snapshot | Yes | Yes (via ContentAgent) |
+| Element Clicks | Yes | Yes (via ContentAgent) |
+| Form Filling | Yes | Yes (via ContentAgent) |
+| Cross-origin | Same-origin only | Any page |
+| Tab Management | No | Yes |
+| Navigation | No | Yes |
+| Screenshots | No | Yes |
+
+## License
+
+Apache-2.0

package/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: agent-browser
+description: Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.
+---
+
+# Browser Automation with agent-browser
+
+## Quick start
+
+```bash
+agent-browser open <url>        # Navigate to page
+agent-browser snapshot -i       # Get interactive elements with refs
+agent-browser click @e1         # Click element by ref
+agent-browser fill @e2 "text"   # Fill input by ref
+agent-browser close             # Close browser
+```
+
+## Core workflow
+
+1. Navigate: `agent-browser open <url>`
+2. Snapshot: `agent-browser snapshot -i` (returns elements with refs like `@e1`, `@e2`)
+3. Interact using refs from the snapshot
+4. Re-snapshot after navigation or significant DOM changes
+
+## Commands
+
+### Navigation
+```bash
+agent-browser open <url>      # Navigate to URL
+agent-browser back            # Go back
+agent-browser forward         # Go forward
+agent-browser reload          # Reload page
+agent-browser close           # Close browser
+```
+
+### Snapshot (page analysis)
+```bash
+agent-browser snapshot        # Full accessibility tree
+agent-browser snapshot -i     # Interactive elements only (recommended)
+agent-browser snapshot -c     # Compact output
+agent-browser snapshot -d 3   # Limit depth to 3
+```
+
+### Interactions (use @refs from snapshot)
+```bash
+agent-browser click @e1           # Click
+agent-browser dblclick @e1        # Double-click
+agent-browser fill @e2 "text"     # Clear and type
+agent-browser type @e2 "text"     # Type without clearing
+agent-browser press Enter         # Press key
+agent-browser press Control+a     # Key combination
+agent-browser hover @e1           # Hover
+agent-browser check @e1           # Check checkbox
+agent-browser uncheck @e1         # Uncheck checkbox
+agent-browser select @e1 "value"  # Select dropdown
+agent-browser scroll down 500     # Scroll page
+agent-browser scrollintoview @e1  # Scroll element into view
+```
+
+### Get information
+```bash
+agent-browser get text @e1        # Get element text
+agent-browser get value @e1       # Get input value
+agent-browser get title           # Get page title
+agent-browser get url             # Get current URL
+```
+
+### Screenshots
+```bash
+agent-browser screenshot          # Screenshot to stdout
+agent-browser screenshot path.png # Save to file
+agent-browser screenshot --full   # Full page
+```
+
+### Wait
+```bash
+agent-browser wait @e1                     # Wait for element
+agent-browser wait 2000                    # Wait milliseconds
+agent-browser wait --text "Success"        # Wait for text
+agent-browser wait --load networkidle      # Wait for network idle
+```
+
+### Semantic locators (alternative to refs)
+```bash
+agent-browser find role button click --name "Submit"
+agent-browser find text "Sign In" click
+agent-browser find label "Email" fill "user@test.com"
+```
+
+## Example: Form submission
+
+```bash
+agent-browser open https://example.com/form
+agent-browser snapshot -i
+# Output shows: textbox "Email" [ref=e1], textbox "Password" [ref=e2], button "Submit" [ref=e3]
+
+agent-browser fill @e1 "user@example.com"
+agent-browser fill @e2 "password123"
+agent-browser click @e3
+agent-browser wait --load networkidle
+agent-browser snapshot -i  # Check result
+```
+
+## Example: Authentication with saved state
+
+```bash
+# Login once
+agent-browser open https://app.example.com/login
+agent-browser snapshot -i
+agent-browser fill @e1 "username"
+agent-browser fill @e2 "password"
+agent-browser click @e3
+agent-browser wait --url "**/dashboard"
+agent-browser state save auth.json
+
+# Later sessions: load saved state
+agent-browser state load auth.json
+agent-browser open https://app.example.com/dashboard
+```
+
+## Sessions (parallel browsers)
+
+```bash
+agent-browser --session test1 open site-a.com
+agent-browser --session test2 open site-b.com
+agent-browser session list
+```
+
+## JSON output (for parsing)
+
+Add `--json` for machine-readable output:
+```bash
+agent-browser snapshot -i --json
+agent-browser get text @e1 --json
+```
+
+## Debugging
+
+```bash
+agent-browser open example.com --headed  # Show browser window
+agent-browser console                    # View console messages
+agent-browser errors                     # View page errors
+```