@dyyz1993/agent-browser 0.9.2 → 0.11.1

package/skills/agent-browser/references/recorder.md

@@ -0,0 +1,319 @@
+# Recorder (Action Recording & Replay)
+
+Record user interactions as structured steps that can be replayed or exported for LLM processing.
+
+**Related**: [commands.md](commands.md) for full command reference, [SKILL.md](../SKILL.md) for quick start.
+
+## Contents
+
+- [Basic Recording](#basic-recording)
+- [Recording Workflow](#recording-workflow)
+- [Supported Actions](#supported-actions)
+- [YAML Output Format](#yaml-output-format)
+- [Replay Feature](#replay-feature)
+- [Use Cases](#use-cases)
+- [Best Practices](#best-practices)
+
+## Basic Recording
+
+```bash
+# Start recording session
+agent-browser recorder start
+
+# Perform actions
+agent-browser open https://example.com
+agent-browser snapshot -i
+agent-browser click @e1
+agent-browser fill @e2 "test input"
+agent-browser select @e3 "option"
+
+# Stop recording and save to file
+agent-browser recorder stop --output session.yaml
+```
+
+## Recording Workflow
+
+The recorder captures all browser interactions including:
+
+1. **Navigation**: Page loads and URL changes
+2. **Input**: Text entry in form fields
+3. **Selection**: Dropdown choices
+4. **Clicks**: Button and link clicks
+5. **Scrolling**: Page scroll events
+6. **Mouse Movement**: Trajectory data for human-like behavior
+
+```bash
+# Example: Complete form submission workflow
+agent-browser recorder start --session form-test
+
+# Navigate to form
+agent-browser open https://example.com/form
+agent-browser snapshot -i
+
+# Fill form fields
+agent-browser fill @e1 "John Doe"
+agent-browser fill @e2 "john@example.com"
+agent-browser select @e3 "United States"
+agent-browser check @e4
+
+# Submit form
+agent-browser click @e5
+agent-browser wait --load networkidle
+
+# Save recording
+agent-browser recorder stop --output form-submission.yaml
+```
+
+## Supported Actions
+
+| Action | Description | Example |
+|--------|-------------|---------|
+| `navigate` | Page navigation | `agent-browser open https://example.com` |
+| `fill` | Text input | `agent-browser fill @e1 "text"` |
+| `select` | Dropdown selection | `agent-browser select @e2 "option"` |
+| `click` | Element click | `agent-browser click @e3` |
+| `check` | Checkbox check | `agent-browser check @e4` |
+| `uncheck` | Checkbox uncheck | `agent-browser uncheck @e5` |
+| `scroll` | Page scroll | `agent-browser scroll down 500` |
+| `trajectory` | Mouse movement | Captured automatically |
+
+## YAML Output Format
+
+The recorder generates a structured YAML file with:
+
+```yaml
+session:
+  id: recorder-1234567890
+  startTime: 19:46:49
+  endTime: 19:48:28
+  steps: 83
+
+pages:
+  - url: https://example.com
+    title: Example Domain
+    firstVisitTime: 19:46:53
+
+steps:
+  - id: step-1234567890
+    time: 19:47:00
+    action: fill
+    selector: "#username"
+    xpath: "//*[@id='username']"
+    value: "testuser"
+
+  - id: step-1234567891
+    time: 19:47:05
+    action: click
+    selector: "#submit-btn"
+    xpath: "//*[@id='submit-btn']"
+
+  - id: step-1234567892
+    time: 19:47:10
+    action: scroll
+    x: 0
+    y: 500
+
+# CLI Commands section contains executable commands
+# for direct replay in terminal
+```
+
+## Replay Feature
+
+Replay recorded interactions from YAML file:
+
+```bash
+# Replay most recent recording
+agent-browser recorder replay
+
+# Replay specific file
+agent-browser recorder replay form-submission.yaml
+
+# Replay with verbose output
+agent-browser recorder replay session.yaml --verbose
+```
+
+The replay feature:
+1. Parses the YAML file
+2. Executes each step in sequence
+3. Handles both navigation and interactions
+4. Supports all recorded action types
+
+## Use Cases
+
+### 1. Test Automation Documentation
+
+Record manual test sessions for documentation:
+
+```bash
+agent-browser recorder start --session checkout-flow
+
+# Perform checkout process
+agent-browser open https://shop.example.com/cart
+agent-browser snapshot -i
+agent-browser fill @e1 "123 Main St"
+agent-browser fill @e2 "New York"
+agent-browser fill @e3 "10001"
+agent-browser click @e4
+agent-browser wait --load networkidle
+
+# Save for documentation
+agent-browser recorder stop --output docs/checkout-flow.yaml
+```
+
+### 2. Regression Testing
+
+Create reusable test scenarios:
+
+```bash
+# Record once
+agent-browser recorder start
+agent-browser open https://app.example.com/login
+agent-browser snapshot -i
+agent-browser fill @e1 "$USERNAME"
+agent-browser fill @e2 "$PASSWORD"
+agent-browser click @e3
+agent-browser recorder stop --output tests/login.yaml
+
+# Replay in CI/CD
+agent-browser recorder replay tests/login.yaml
+```
+
+### 3. Workflow Automation
+
+Capture complex workflows for automation:
+
+```bash
+# Record multi-step workflow
+agent-browser recorder start
+agent-browser open https://dashboard.example.com
+agent-browser snapshot -i
+agent-browser click @e1  # Navigate to reports
+agent-browser click @e2  # Select date range
+agent-browser click @e3  # Export CSV
+agent-browser recorder stop --output workflows/export-data.yaml
+```
+
+### 4. Debugging & Analysis
+
+Record sessions for debugging:
+
+```bash
+agent-browser recorder start --session debug-$(date +%s)
+
+# Run problematic workflow
+agent-browser open https://example.com
+# ... interactions ...
+
+# Save for analysis
+agent-browser recorder stop --output debug/session.yaml
+```
+
+## Best Practices
+
+### 1. Use Session Names
+
+```bash
+# Good: Descriptive session names
+agent-browser recorder start --session user-registration
+agent-browser recorder start --session checkout-payment
+agent-browser recorder start --session search-functionality
+
+# Avoid: Generic names
+agent-browser recorder start --session test1
+agent-browser recorder start --session recording
+```
+
+### 2. Add Wait Times for Stability
+
+```bash
+agent-browser recorder start
+
+# Add waits after critical actions
+agent-browser click @e1
+agent-browser wait --load networkidle  # Wait for page load
+
+agent-browser fill @e2 "text"
+agent-browser wait 1000  # Wait for dynamic content
+
+agent-browser click @e3
+```
+
+### 3. Use Snapshots for Ref Stability
+
+```bash
+agent-browser recorder start
+
+# Always snapshot before interactions
+agent-browser snapshot -i
+agent-browser click @e1
+
+# Re-snapshot after navigation
+agent-browser wait --load networkidle
+agent-browser snapshot -i
+agent-browser click @e2
+```
+
+### 4. Organize Recordings
+
+```bash
+# Create organized directory structure
+recordings/
+├── tests/
+│   ├── login.yaml
+│   ├── registration.yaml
+│   └── checkout.yaml
+├── workflows/
+│   ├── data-export.yaml
+│   └── report-generation.yaml
+└── docs/
+    ├── user-guide.yaml
+    └── api-demo.yaml
+```
+
+### 5. Review Generated Commands
+
+The YAML file includes a CLI Commands section at the end with executable commands. Review these commands to:
+
+- Verify the captured selectors
+- Check for redundant steps
+- Identify opportunities for optimization
+- Ensure actions are in correct order
+
+## Advanced Features
+
+### Session-Based Recording
+
+```bash
+# Record with specific session
+agent-browser recorder start --session my-test --timeout 60000
+
+# Use session for all commands
+agent-browser open https://example.com --session my-test
+agent-browser snapshot -i --session my-test
+agent-browser click @e1 --session my-test
+
+# Stop recording
+agent-browser recorder stop --output my-test.yaml --session my-test
+```
+
+### Timeout Configuration
+
+```bash
+# Set recording timeout (default: 60 seconds)
+agent-browser recorder start --timeout 120000  # 2 minutes
+
+# Useful for long-running workflows
+```
+
+## Limitations
+
+- Refs (`@e1`, `@e2`) are session-specific and not portable
+- Convert to CSS selectors for cross-session reuse
+- Some dynamic content may require additional wait handling
+- Replay requires same page structure as recording
+
+## See Also
+
+- [snapshot-refs.md](snapshot-refs.md) - Understanding refs and their lifecycle
+- [authentication.md](authentication.md) - Recording login flows
+- [video-recording.md](video-recording.md) - Video recording for debugging

package/skills/agent-browser/references/viewer-mode.md

@@ -0,0 +1,148 @@
+# Viewer / Streaming Mode
+
+## Overview
+
+The viewer mode provides a **real-time visual remote browser interface**. It streams browser frames (JPEG/WebP) over WebSocket and forwards user input (mouse, keyboard, touch) back to the daemon. This enables:
+
+- **Remote debugging** — see what the browser sees in real time
+- **Mobile device control** — operate a desktop browser from your phone
+- **Presentation/demo** — show browser activity to an audience
+- **Collaboration** — share a browser session with others
+
+## Starting the Viewer
+
+```bash
+# Prerequisite: have a browser session running
+agent-browser open https://example.com
+
+# Start viewer (opens URL in default browser)
+agent-browser viewer
+
+# Get connection details as JSON (for scripting/embedding)
+agent-browser viewer --json
+# Output: {"url":"http://localhost:5005/view?session=default","ws":"ws://...","port":5005}
+```
+
+## Viewer URL Parameters
+
+| Parameter          | Description                            |
+| ------------------ | -------------------------------------- |
+| `?session=<id>`    | Connect to a specific named session    |
+| `?instanceId=<id>` | Connect to a specific browser instance |
+
+## Architecture
+
+```
+┌─────────────┐    IPC     ┌───────────────────┐   WebSocket   ┌──────────┐
+│  Browser    │ ───────→ │  Daemon Process   │ ←────────────→ │  Viewer  │
+│  (Playwright) │           │  (:5000 socket)   │               │  (Browser) │
+└─────────────┘           └────────┬─────────┘               └──────────┘
+                             │
+                    standalone HTTP+WS server (:5005)
+                    serves viewer.html + proxies messages
+```
+
+**Data flow:**
+
+1. **Frames**: Browser -> Daemon -> Standalone Server -> Viewer (binary JPEG/WebP via WS)
+2. **Input**: Viewer -> Standalone Server -> Daemon -> Browser (JSON messages)
+
+## Viewer Page Features
+
+### Desktop Mode (PC/Mac)
+
+| Feature      | Description                                                         |
+| ------------ | ------------------------------------------------------------------- |
+| Screen area  | Shows streamed frame, click/drag/scroll sends input to remote       |
+| Toolbar      | URL bar, connection status, quality badge, record button            |
+| Hidden input | Invisible capture field for keyboard events (auto-focused on click) |
+| Cursor       | Red dot showing remote mouse position                               |
+
+### Mobile Mode (Touch Device)
+
+Automatically activates on touch devices. See [mobile-viewer.md](mobile-viewer.md) for full details.
+
+| Feature          | Description                                         |
+| ---------------- | --------------------------------------------------- |
+| Touchpad         | Bottom gesture area for cursor simulation           |
+| Input Panel      | Text input popup when tapping remote input fields   |
+| Keyboard toolbar | Virtual keys: Tab, Arrows, Enter, Backspace, Escape |
+| IME support      | Chinese/Japanese composition (pinyin, kana, etc.)   |
+
+## Element Selector / Crop Mode
+
+Crop the video stream to a specific DOM element's bounds:
+
+```bash
+# Via viewer UI: click element selector button, then click target element
+# The stream is cropped to that element's rectangle
+```
+
+When element mode is active:
+
+- Server crops frames to element bounds using Sharp
+- Mouse coordinates auto-map to element-local space
+- Falls back to "degraded mode" (full page) if element not found or disappears
+- `deviceWidth`/`deviceHeight` in metadata reflect element dimensions
+
+Use cases:
+
+- Focus testing on a specific component
+- Recording interactions within a widget
+- Bandwidth savings (only stream the element, not full page)
+
+## Message Types (Viewer <-> Server)
+
+### Server → Viewer (over WebSocket)
+
+| Type            | Purpose                                                            |
+| --------------- | ------------------------------------------------------------------ |
+| `frame`         | Binary frame data with metadata (dimensions, format, element info) |
+| `status`        | Connection status, viewport changes                                |
+| `navigation`    | URL/title changes                                                  |
+| `input_focused` | Remote element received focus → triggers input panel (mobile)      |
+| `input_value`   | Remote input value changed                                         |
+| `input_blur`    | Remote element lost focus                                          |
+
+### Viewer → Server (over WebSocket)
+
+| Type                   | Purpose                                             |
+| ---------------------- | --------------------------------------------------- |
+| `input_mouse`          | Mouse move/press/release/wheel                      |
+| `input_keyboard`       | Key down/up with modifiers                          |
+| `input_fill`           | Full text value sync (mobile input panel)           |
+| `input_blur_element`   | Blur remote element (mobile input commit)           |
+| `keyboard_insert_text` | Character-by-character insert (desktop hiddenInput) |
+| `user_activity`        | Keep-alive signal (resumes streaming if paused)     |
+| `selector_element`     | Request crop to specific element                    |
+
+## Troubleshooting
+
+### Black screen
+
+- Check daemon is running: `agent-browser status`
+- Verify browser launched: `agent-browser open https://example.com` should work first
+
+### Connection refused
+
+- The viewer command auto-starts the standalone server on port 5005
+- If port conflicts, check: `lsof -i :5005`
+- Kill stale process: `kill $(lsof -t -i :5005)`
+
+### Laggy updates
+
+- Frame compression is JPEG by default (adjustable)
+- Quality badge shows current state: "interacting" / "static" / "compressed"
+- Network latency between viewer and server affects frame rate
+
+### Element not found (degraded mode)
+
+- Yellow toast appears: "Element not found, showing full page"
+- Element may have been removed by SPA navigation or animation
+- Re-select the element or exit selector mode
+
+### Viewer shows but no frame
+
+- Check daemon log: `~/.agent-browser/default.log`
+- Look for "Browser not launched" errors
+- Ensure `agent-browser open` was called before `agent-browser viewer`

package/skills/agent-browser/templates/api-interception.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
 # API Interception Template - Passively capture API responses
+set -euo pipefail
 # Usage: ./api-interception.sh [target_url] [output_file]
 
 TARGET_URL="${1:-https://example.com/user/profile}"
@@ -12,7 +13,8 @@ sleep 1
 
 echo ""
 echo "=== 2. Open blank page ==="
-export https_proxy=http://127.0.0.1:7890
+# Optional: set PROXY_URL if using a proxy
+export https_proxy=${PROXY_URL:-}
 agent-browser open "about:blank"
 sleep 1
 

package/skills/agent-browser/templates/data-extraction.sh

@@ -34,7 +34,8 @@ case "$MODE" in
   api)
     echo ""
     echo "=== 2. API Interception Mode ==="
-    export https_proxy=http://127.0.0.1:7890
+    # Optional: set PROXY_URL if using a proxy
+export https_proxy=${PROXY_URL:-}
     agent-browser open "about:blank"
     sleep 1
     
@@ -64,7 +65,8 @@ case "$MODE" in
   scroll)
     echo ""
     echo "=== 2. Infinite Scroll Mode ==="
-    export https_proxy=http://127.0.0.1:7890
+    # Optional: set PROXY_URL if using a proxy
+export https_proxy=${PROXY_URL:-}
     agent-browser open "$TARGET_URL"
     sleep 2
     
@@ -125,7 +127,8 @@ print(json.dumps(unique, ensure_ascii=False))
   js)
     echo ""
     echo "=== 2. JS Variable Extraction Mode ==="
-    export https_proxy=http://127.0.0.1:7890
+    # Optional: set PROXY_URL if using a proxy
+export https_proxy=${PROXY_URL:-}
     agent-browser open "$TARGET_URL"
     sleep 3
     
@@ -153,7 +156,8 @@ print(json.dumps(unique, ensure_ascii=False))
   dom|*)
     echo ""
     echo "=== 2. DOM Extraction Mode ==="
-    export https_proxy=http://127.0.0.1:7890
+    # Optional: set PROXY_URL if using a proxy
+export https_proxy=${PROXY_URL:-}
     agent-browser open "$TARGET_URL"
     sleep 2
     

package/skills/agent-browser/templates/form-automation.sh

@@ -1,32 +1,27 @@
-#!/bin/bash
+#!/usr/bin/env bash
 # Template: Form Automation Workflow
 # Purpose: Fill and submit web forms with validation
 # Usage: ./form-automation.sh <form-url>
 #
-# This template demonstrates the snapshot-interact-verify pattern:
-# 1. Navigate to form
-# 2. Snapshot to get element refs
-# 3. Fill fields using refs
-# 4. Submit and verify result
-#
-# Customize: Update the refs (@e1, @e2, etc.) based on your form's snapshot output
+# Demonstrates: snapshot -> interact -> verify pattern
 
 set -euo pipefail
 
 FORM_URL="${1:?Usage: $0 <form-url>}"
+SESSION="form-$(date +%s)"
 
-echo "Form automation: $FORM_URL"
+echo "=== Form Automation: $FORM_URL ==="
 
 # Step 1: Navigate to form
-agent-browser open "$FORM_URL"
-agent-browser wait --load networkidle
+agent-browser --session "$SESSION" open "$FORM_URL"
+agent-browser --session "$SESSION" wait --load networkidle
 
 # Step 2: Snapshot to discover form elements
 echo ""
 echo "Form structure:"
-agent-browser snapshot -i
+agent-browser --session "$SESSION" snapshot -i
 
-# Step 3: Fill form fields (customize these refs based on snapshot output)
+# Step 3: Fill form fields (customize refs based on snapshot output above)
 #
 # Common field types:
 #   agent-browser fill @e1 "John Doe"           # Text input
@@ -34,27 +29,27 @@ agent-browser snapshot -i
 #   agent-browser fill @e3 "SecureP@ss123"      # Password input
 #   agent-browser select @e4 "Option Value"     # Dropdown
 #   agent-browser check @e5                     # Checkbox
-#   agent-browser click @e6                     # Radio button
+#   agent-browser click @e6                     # Radio button / Submit button
 #   agent-browser fill @e7 "Multi-line text"   # Textarea
 #   agent-browser upload @e8 /path/to/file.pdf # File upload
 #
 # Uncomment and modify:
-# agent-browser fill @e1 "Test User"
-# agent-browser fill @e2 "test@example.com"
-# agent-browser click @e3  # Submit button
+# agent-browser --session "$SESSION" fill @e1 "Test User"
+# agent-browser --session "$SESSION" fill @e2 "test@example.com"
+# agent-browser --session "$SESSION" click @e3  # Submit button
 
-# Step 4: Wait for submission
-# agent-browser wait --load networkidle
-# agent-browser wait --url "**/success"  # Or wait for redirect
+# Step 4: Wait for submission to complete
+agent-browser --session "$SESSION" wait --load networkidle
+# agent-browser --session "$SESSION" wait --url "**/success"  # Or wait for redirect
 
 # Step 5: Verify result
 echo ""
 echo "Result:"
-agent-browser get url
-agent-browser snapshot -i
+agent-browser --session "$SESSION" get url
+agent-browser --session "$SESSION" snapshot -i
 
 # Optional: Capture evidence
-agent-browser screenshot /tmp/form-result.png
+agent-browser --session "$SESSION" screenshot /tmp/form-result.png
 echo "Screenshot saved: /tmp/form-result.png"
 
 # Cleanup