btcp-browser-agent 0.1.0

package/examples/chrome-extension/tests/README.md

@@ -0,0 +1,206 @@
+# Chrome Extension Scenario Tests
+
+This directory contains demonstration scripts that showcase how AI agents would interact with the BTCP Browser Agent API in real-world scenarios.
+
+## Available Scenarios
+
+### 1. Google Search → GitHub Star (`scenario-google-to-github-star.ts`)
+
+A complete workflow demonstrating AI agent reasoning patterns:
+
+**Workflow:**
+1. Navigate to Google
+2. Search for "btcp-cowork"
+3. Find and click the first GitHub link in results
+4. Navigate to the repository
+5. Attempt to star the repository
+
+**What it demonstrates:**
+- AI reasoning simulation with detailed logging
+- Snapshot-based page understanding
+- Element selection strategies using accessibility tree
+- Error recovery with fallback approaches
+- Multi-step navigation and validation
+- Authentication gate detection
+
+## Running the Scenarios
+
+### Prerequisites
+
+1. **Install Dependencies**
+   ```bash
+   cd examples/chrome-extension
+   npm install
+   ```
+
+2. **Build the Extension**
+   ```bash
+   npm run build
+   ```
+
+3. **Load Extension in Chrome**
+   - Open Chrome and navigate to `chrome://extensions`
+   - Enable "Developer mode" (toggle in top right)
+   - Click "Load unpacked"
+   - Select the `examples/chrome-extension/dist/` directory
+
+4. **Create a Session**
+   - Click the extension icon in Chrome toolbar
+   - Click "Start Session" button
+   - A new tab group will be created
+
+### Running the Demo
+
+With the extension loaded and a session active:
+
+```bash
+cd examples/chrome-extension
+npm run demo:scenario
+```
+
+### Expected Output
+
+The script will output detailed logs showing AI reasoning at each step:
+
+```
+🤔 AI Agent: Starting Google → GitHub → Star workflow demonstration
+
+[Step 1/15] Initialize connection to browser extension
+✨ Success: Client connected
+
+[Step 2/15] Navigate to Google
+🤔 AI Agent: Need to navigate to google.com to start search
+🎯 Action: Navigating to https://www.google.com...
+✅ Navigation complete
+
+[Step 3/15] Take snapshot to analyze page structure
+🤔 AI Agent: Taking snapshot to identify interactive elements
+🎯 Action: Calling snapshot API...
+📊 Verification: Snapshot captured: 156 elements found
+
+[Step 4/15] Locate search input field
+🔍 Analyzing: Looking for search box (role=searchbox or combobox)
+✅ Found: @ref:12 - role='combobox' name='Search'
+
+...
+```
+
+## Understanding the Output
+
+The demo uses emoji prefixes to indicate different types of information:
+
+- 🤔 **Thinking**: AI agent's reasoning process
+- 🔍 **Analyzing**: Element search criteria
+- ✅ **Found**: Successfully located element
+- 🎯 **Action**: Executing a command
+- 📊 **Verification**: Validating results
+- ❌ **Error**: Something went wrong
+- ⚠️ **Warning**: Non-critical issue or fallback triggered
+- ✨ **Success**: Step completed successfully
+
+## Common Issues
+
+### "Could not find search input"
+
+Google's page structure varies by region and personalization. The script includes fallback strategies, but may need adjustment for your specific Google layout.
+
+**Solution**: The script will try multiple approaches automatically. Check the logs to see which strategy worked.
+
+### "No GitHub links found in search results"
+
+This can happen if:
+- Google blocks automated searches
+- Search results don't include GitHub repositories
+- Results are personalized differently
+
+**Solution**: Try running the script again, or manually verify that searching "btcp-cowork" on Google returns GitHub results.
+
+### "Redirected to login page"
+
+GitHub requires authentication to star repositories.
+
+**Solution**: This is expected behavior. The script will detect the login redirect and explain the situation. In a real AI agent, this would trigger an OAuth flow or use stored credentials.
+
+### "Error: connect ECONNREFUSED"
+
+The extension is not running or no session is active.
+
+**Solution**:
+1. Verify the extension is loaded in Chrome
+2. Click the extension icon
+3. Click "Start Session"
+4. Run the script again
+
+## API Quality Insights
+
+This scenario helps evaluate the BTCP API for AI agent use cases:
+
+### Strengths
+✅ **Clear accessibility tree**: Snapshot provides semantic element information
+✅ **Stable references**: `@ref:N` selectors work reliably within a session
+✅ **Simple API**: Navigation and interaction methods are intuitive
+✅ **Error messages**: Informative feedback when operations fail
+
+### Areas for Improvement
+⚠️ **Complex pages**: Some dynamic content needs multiple snapshot strategies
+⚠️ **Authentication**: OAuth flows require additional handling
+⚠️ **Timing**: Some actions need explicit wait times (could use smart waiting)
+⚠️ **Ref invalidation**: References become stale after navigation (expected, but needs handling)
+
+## Writing Your Own Scenarios
+
+To create a new scenario test:
+
+1. **Create a new file** in this directory (e.g., `scenario-form-filling.ts`)
+
+2. **Import the client**:
+   ```typescript
+   import { createClient } from '../../../packages/extension/src/index.js';
+   ```
+
+3. **Structure your workflow**:
+   ```typescript
+   async function main() {
+     const client = createClient();
+
+     // Step 1: Navigate
+     await client.navigate('https://example.com');
+
+     // Step 2: Understand page
+     const snapshot = await client.snapshot({ format: 'tree' });
+
+     // Step 3: Find and interact with elements
+     const button = findElement(snapshot.tree, { role: 'button', name: 'Submit' });
+     await client.click(button);
+
+     // Step 4: Verify outcome
+     const newUrl = await client.getUrl();
+     console.log('Success:', newUrl.includes('success'));
+   }
+
+   main();
+   ```
+
+4. **Add logging** to show AI reasoning at each step
+
+5. **Add to package.json**:
+   ```json
+   "scripts": {
+     "demo:your-scenario": "tsx tests/scenario-your-name.ts"
+   }
+   ```
+
+## Contributing
+
+When adding new scenarios, please:
+- Include detailed AI reasoning logs
+- Handle errors gracefully with fallbacks
+- Document what the scenario tests/demonstrates
+- Update this README with usage instructions
+- Consider edge cases and authentication requirements
+
+## Further Reading
+
+- [BTCP API Documentation](../../CLAUDE.md)
+- [Extension Architecture](../../../packages/extension/README.md)
+- [Core Actions Reference](../../../packages/core/README.md)

package/examples/chrome-extension/tests/scenario-google-to-github-star.ts

@@ -0,0 +1,380 @@
+/**
+ * Real-World AI Agent Scenario: Google Search → GitHub Repository → Star
+ *
+ * This demo script mimics how an AI agent would reason through a complete browser workflow:
+ * 1. Navigate to Google
+ * 2. Search for "btcp-cowork"
+ * 3. Find and click the first GitHub link in results
+ * 4. Attempt to star the repository
+ *
+ * The script demonstrates:
+ * - AI reasoning patterns (element selection, verification, error handling)
+ * - Snapshot-based navigation (accessibility tree parsing)
+ * - Graceful error recovery
+ * - Step-by-step validation
+ *
+ * Prerequisites:
+ * - Chrome extension must be loaded and running
+ * - Extension must have a session created
+ *
+ * Run with: npm run demo:scenario
+ */
+
+import { createClient } from '../../../packages/extension/src/index.js';
+
+// Utility for delays (simulating think time)
+const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
+
+// Console formatting helpers
+const log = {
+  thinking: (msg: string) => console.log(`\n🤔 AI Agent: ${msg}`),
+  analyzing: (msg: string) => console.log(`🔍 Analyzing: ${msg}`),
+  found: (msg: string) => console.log(`✅ Found: ${msg}`),
+  action: (msg: string) => console.log(`🎯 Action: ${msg}`),
+  verify: (msg: string) => console.log(`📊 Verification: ${msg}`),
+  error: (msg: string) => console.log(`❌ Error: ${msg}`),
+  warning: (msg: string) => console.log(`⚠️  Warning: ${msg}`),
+  success: (msg: string) => console.log(`✨ Success: ${msg}`),
+  step: (step: number, total: number, msg: string) => console.log(`\n[Step ${step}/${total}] ${msg}`),
+};
+
+/**
+ * Simulates AI reasoning to find an element in the snapshot tree
+ * Returns the @ref:N selector for the found element
+ */
+function findElement(tree: string, criteria: {
+  role?: string;
+  name?: string;
+  nameContains?: string;
+  type?: string;
+}): string | null {
+  const lines = tree.split('\n');
+
+  for (const line of lines) {
+    // Parse line format: "@ref:N role='...' name='...' ..."
+    const refMatch = line.match(/@ref:(\d+)/);
+    if (!refMatch) continue;
+
+    const ref = `@ref:${refMatch[1]}`;
+
+    // Check all criteria
+    let matches = true;
+
+    if (criteria.role) {
+      const roleMatch = line.match(/role='([^']+)'/);
+      if (!roleMatch || roleMatch[1] !== criteria.role) matches = false;
+    }
+
+    if (criteria.name) {
+      const nameMatch = line.match(/name='([^']+)'/);
+      if (!nameMatch || nameMatch[1] !== criteria.name) matches = false;
+    }
+
+    if (criteria.nameContains) {
+      const nameMatch = line.match(/name='([^']+)'/);
+      if (!nameMatch || !nameMatch[1].toLowerCase().includes(criteria.nameContains.toLowerCase())) {
+        matches = false;
+      }
+    }
+
+    if (criteria.type) {
+      const typeMatch = line.match(/type='([^']+)'/);
+      if (!typeMatch || typeMatch[1] !== criteria.type) matches = false;
+    }
+
+    if (matches) {
+      log.found(`${ref} - ${line.trim()}`);
+      return ref;
+    }
+  }
+
+  return null;
+}
+
+/**
+ * Find first link containing specific domain
+ */
+function findLinkByDomain(tree: string, domain: string): string | null {
+  const lines = tree.split('\n');
+
+  for (const line of lines) {
+    if (!line.includes("role='link'")) continue;
+
+    const refMatch = line.match(/@ref:(\d+)/);
+    if (!refMatch) continue;
+
+    // Check if line contains the domain (in name or other attributes)
+    if (line.toLowerCase().includes(domain.toLowerCase())) {
+      const ref = `@ref:${refMatch[1]}`;
+      log.found(`${ref} - GitHub link: ${line.trim()}`);
+      return ref;
+    }
+  }
+
+  return null;
+}
+
+async function main() {
+  const TOTAL_STEPS = 15;
+  let currentStep = 0;
+
+  log.thinking('Starting Google → GitHub → Star workflow demonstration');
+  console.log('This script simulates AI agent reasoning patterns for browser automation\n');
+
+  try {
+    // Step 1: Initialize client
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Initialize connection to browser extension');
+    const client = createClient();
+    log.success('Client connected');
+    await sleep(500);
+
+    // Step 2: Navigate to Google
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Navigate to Google');
+    log.thinking('Need to navigate to google.com to start search');
+    log.action('Navigating to https://www.google.com...');
+
+    await client.navigate('https://www.google.com');
+    log.success('Navigation complete');
+    await sleep(1000); // Wait for page to settle
+
+    // Step 3: Take snapshot to understand page structure
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Take snapshot to analyze page structure');
+    log.thinking('Taking snapshot to identify interactive elements');
+    log.action('Calling snapshot API...');
+
+    const snapshot1 = await client.snapshot({ format: 'tree' });
+    log.verify(`Snapshot captured: ${snapshot1.split('\n').length} elements found`);
+    await sleep(500);
+
+    // Step 4: Find search input
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Locate search input field');
+    log.analyzing('Looking for search box (role=searchbox or combobox)');
+
+    let searchInput = findElement(snapshot1, { role: 'combobox' });
+    if (!searchInput) {
+      searchInput = findElement(snapshot1, { role: 'searchbox' });
+    }
+
+    if (!searchInput) {
+      log.error('Could not find search input on Google homepage');
+      log.warning('This might be due to Google\'s dynamic content or region-specific layout');
+      // Try alternative approach: look for input with name containing "search" or "q"
+      log.analyzing('Trying alternative strategy: looking for textbox with search-related name');
+      searchInput = findElement(snapshot1, { role: 'textbox', nameContains: 'search' });
+    }
+
+    if (!searchInput) {
+      throw new Error('Unable to locate search input field');
+    }
+
+    await sleep(500);
+
+    // Step 5: Type search query
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Type search query into input');
+    log.thinking('Need to type "btcp-cowork" into the search box');
+    log.action(`Typing "btcp-cowork" into ${searchInput}...`);
+
+    await client.type(searchInput, 'btcp-cowork');
+    log.success('Query typed successfully');
+    await sleep(500);
+
+    // Step 6: Find and click search button
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Locate and click search button');
+    log.analyzing('Looking for search submit button (role=button, name contains "search")');
+
+    const searchButton = findElement(snapshot1, { role: 'button', nameContains: 'search' });
+
+    if (!searchButton) {
+      log.warning('Search button not found, trying Enter key instead');
+      log.action('Pressing Enter key to submit search...');
+      await client.execute({
+        id: crypto.randomUUID(),
+        action: 'press',
+        key: 'Enter',
+      });
+    } else {
+      log.action(`Clicking search button ${searchButton}...`);
+      await client.click(searchButton);
+    }
+
+    log.success('Search submitted');
+    await sleep(2000); // Wait for search results to load
+
+    // Step 7: Wait for results page to load
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Wait for search results to load');
+    log.thinking('Giving page time to load search results');
+
+    const currentUrl = await client.getUrl();
+    log.verify(`Current URL: ${currentUrl}`);
+
+    if (!currentUrl.includes('google.com/search')) {
+      log.warning('URL does not appear to be a search results page');
+    }
+    await sleep(1000);
+
+    // Step 8: Take snapshot of results page
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Snapshot search results page');
+    log.action('Taking snapshot of search results...');
+
+    const snapshot2 = await client.snapshot({ format: 'tree' });
+    log.verify(`Results snapshot captured: ${snapshot2.split('\n').length} elements found`);
+    await sleep(500);
+
+    // Step 9: Find first GitHub link
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Locate first GitHub link in results');
+    log.analyzing('Searching for links containing "github.com"');
+    log.thinking('AI reasoning: GitHub links are likely to contain "github" in the URL or link text');
+
+    const githubLink = findLinkByDomain(snapshot2, 'github.com');
+
+    if (!githubLink) {
+      log.error('No GitHub links found in search results');
+      log.warning('Possible reasons:');
+      console.log('  - Search results may not include GitHub repositories');
+      console.log('  - Google may have blocked automated searches');
+      console.log('  - Results structure may differ from expected format');
+      throw new Error('Cannot proceed without GitHub link');
+    }
+
+    await sleep(500);
+
+    // Step 10: Click GitHub link
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Navigate to GitHub repository');
+    log.action(`Clicking GitHub link ${githubLink}...`);
+
+    await client.click(githubLink);
+    log.success('Clicked GitHub link, waiting for page load...');
+    await sleep(3000); // Wait for GitHub page to load
+
+    // Step 11: Verify we're on GitHub
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Verify navigation to GitHub');
+
+    const githubUrl = await client.getUrl();
+    log.verify(`Current URL: ${githubUrl}`);
+
+    if (!githubUrl.includes('github.com')) {
+      log.warning('URL does not appear to be github.com');
+    } else {
+      log.success('Successfully navigated to GitHub');
+    }
+
+    await sleep(1000);
+
+    // Step 12: Take snapshot of GitHub repo page
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Snapshot GitHub repository page');
+    log.action('Taking snapshot of repository page...');
+
+    const snapshot3 = await client.snapshot({ format: 'tree' });
+    log.verify(`GitHub page snapshot: ${snapshot3.split('\n').length} elements found`);
+    await sleep(500);
+
+    // Step 13: Find star button
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Locate star button');
+    log.analyzing('Looking for Star/Unstar button (role=button, name contains "star")');
+    log.thinking('AI reasoning: Star button typically has role=button and name includes "star" or "unstar"');
+
+    let starButton = findElement(snapshot3, { role: 'button', nameContains: 'star' });
+
+    if (!starButton) {
+      log.warning('Star button not found in standard format');
+      log.analyzing('Trying alternative: looking for any element with "star" in name');
+
+      // Try to find any element with "star" in it
+      const lines = snapshot3.split('\n');
+      for (const line of lines) {
+        if (line.toLowerCase().includes('star')) {
+          const refMatch = line.match(/@ref:(\d+)/);
+          if (refMatch) {
+            starButton = `@ref:${refMatch[1]}`;
+            log.found(`Possible star element: ${line.trim()}`);
+            break;
+          }
+        }
+      }
+    }
+
+    if (!starButton) {
+      log.error('Could not locate star button');
+      log.warning('Possible reasons:');
+      console.log('  - User may not be logged into GitHub');
+      console.log('  - Page structure may differ from expected format');
+      console.log('  - Repository may have restricted star functionality');
+      throw new Error('Cannot proceed without star button');
+    }
+
+    await sleep(500);
+
+    // Step 14: Click star button
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Attempt to star the repository');
+    log.thinking('Clicking star button - this may require GitHub login');
+    log.action(`Clicking star button ${starButton}...`);
+
+    await client.click(starButton);
+    log.success('Star button clicked');
+    await sleep(2000);
+
+    // Step 15: Verify outcome
+    currentStep++;
+    log.step(currentStep, TOTAL_STEPS, 'Verify final state');
+
+    const finalUrl = await client.getUrl();
+    log.verify(`Final URL: ${finalUrl}`);
+
+    if (finalUrl.includes('login')) {
+      log.warning('Redirected to login page - authentication required to star repositories');
+      log.thinking('AI reasoning: In a real scenario, the agent would need to handle OAuth flow or use stored credentials');
+    } else if (finalUrl.includes('github.com')) {
+      log.success('Still on GitHub page - star action may have succeeded');
+      log.thinking('Taking final snapshot to verify star state...');
+
+      const finalSnapshot = await client.snapshot({ format: 'tree' });
+      const hasUnstar = finalSnapshot.toLowerCase().includes('unstar');
+
+      if (hasUnstar) {
+        log.success('✨ Repository successfully starred! (Found "Unstar" button)');
+      } else {
+        log.verify('Star action completed, but state verification inconclusive');
+      }
+    }
+
+    console.log('\n' + '='.repeat(80));
+    log.success('🎉 Demo workflow completed successfully!');
+    console.log('='.repeat(80));
+
+    console.log('\n📊 Workflow Summary:');
+    console.log(`   • Navigated to Google`);
+    console.log(`   • Searched for "btcp-cowork"`);
+    console.log(`   • Found and clicked GitHub link`);
+    console.log(`   • Attempted to star repository`);
+    console.log(`   • Total steps executed: ${currentStep}/${TOTAL_STEPS}`);
+
+    console.log('\n💡 API Quality Assessment:');
+    console.log('   ✅ Snapshot API provides clear accessibility tree');
+    console.log('   ✅ Element refs (@ref:N) are stable and easy to use');
+    console.log('   ✅ Navigation and interaction methods work reliably');
+    console.log('   ⚠️  Some complex page structures need fallback strategies');
+    console.log('   ⚠️  Authentication flows require additional handling');
+
+  } catch (error) {
+    log.error(`Workflow failed: ${error instanceof Error ? error.message : String(error)}`);
+    console.error(error);
+    process.exit(1);
+  }
+}
+
+// Run the demo
+main();

package/examples/chrome-extension/tsconfig.json

@@ -0,0 +1,14 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "outDir": "dist",
+    "rootDir": "src",
+    "types": ["chrome"]
+  },
+  "include": ["src/**/*"]
+}