@caoscompanybr/merlin 3.5.0

package/.merlin-core/skills/domain/browser-automation/SKILL.md

@@ -0,0 +1,311 @@
+---
+name: browser-automation
+description: Comprehensive browser automation skill — Chrome/Chromium control via CDP (agent-browser CLI), Playwright (Python async/sync), persistent sessions, anti-detection, platform automation, form filling, screenshots, authentication flows, and visual testing. Combines Vercel's agent-browser best practices with production patterns from an internal Chrome-automation engine and MCP Playwright integration.
+license: Apache-2.0
+metadata:
+  author: merlin-framework (sources — vercel-labs/agent-browser MIT, internal Chrome-automation production patterns)
+  version: "1.0.0"
+  auto_activate: [ browser-automation, chrome-automation, playwright-agent, headless-browser, agent-browser-cli, ]
+  file_triggers: [**/chrome_automation*, **/*automator*, **/*playwright*, **/agent-browser*]
+  tool_reminders: [Browser automation that logs into third-party services is a HIGH-stake action — require explicit human approval, Use the browser-takeover protocol [ASK: BROWSER_TAKEOVER] for login, payment, 2FA, and CAPTCHA — NEVER automate these, Always implement rate limiting and daily quotas when automating platform interactions, Anti-detection measures are mandatory for production — random delays, mouse offsets, disable AutomationControlled, Prefer MCP Playwright when available (real-time, interactive); fall back to agent-browser CLI for scripted automation]
+---
+
+## Browser Automation — Domain Skill
+
+Expert knowledge for browser automation across the Merlin ecosystem. Three layers of capability, unified into one skill.
+
+### When This Activates
+
+This skill activates when working on browser automation, web scraping, Chrome control, Playwright tests, or platform interactions. Current ecosystem:
+
+- **An internal Chrome-automation engine** — Production Chrome automation (Freepik, Gemini, Meta AI platform automators)
+- **<client project>** — QA visual verification via MCP Playwright
+- **Any Merlin project** — E2E testing, form automation, screenshots, data extraction
+
+### Architecture: Three Layers
+
+```
+Layer 1: MCP Playwright (Real-time, Interactive)
+    ├── @anthropic-ai/mcp-playwright server
+    ├── Direct browser control from Claude Code
+    ├── Network inspection, screenshots, DOM queries
+    └── Config: .mcp.json at project root
+
+Layer 2: agent-browser CLI (Scripted Automation)
+    ├── Chrome DevTools Protocol (CDP)
+    ├── Snapshot → Reference → Interact cycle
+    ├── Session management, state persistence
+    └── Install: npm i -g agent-browser && agent-browser install
+
+Layer 3: Playwright Python (Test Suites + Production)
+    ├── playwright.async_api (internal Chrome-automation engine, production)
+    ├── playwright.sync_api (Merlin webapp-testing skill)
+    ├── Persistent profiles, anti-detection
+    └── Platform-specific automators with rate limiting
+```
+
+### Decision Matrix: Which Layer to Use
+
+| Scenario                                   | Use                       | Why                                                |
+| ------------------------------------------ | ------------------------- | -------------------------------------------------- |
+| Interactive debugging in Claude Code       | MCP Playwright            | Real-time, visual feedback                         |
+| Quick form fill or data extraction         | agent-browser CLI         | Fast, scriptable, no code needed                   |
+| E2E test suite                             | Playwright Python (sync)  | Standard test framework patterns                   |
+| Production platform automation             | Playwright Python (async) | Persistent sessions, anti-detection, rate limiting |
+| Visual regression testing                  | webapp-testing skill      | Screenshot comparison built-in                     |
+| Sensitive operations (login, payment, 2FA) | browser-takeover protocol | Human-in-the-loop, NEVER automate                  |
+
+---
+
+## Layer 1: MCP Playwright (Interactive)
+
+Configured in `.mcp.json` at project root:
+
+```json
+{
+  "mcpServers": {
+    "browser": {
+      "command": "npx",
+      "args": ["@anthropic-ai/mcp-playwright"]
+    }
+  }
+}
+```
+
+**Tools available when MCP is active:**
+
+- `browser:navigate` — Go to URL
+- `browser:screenshot` — Capture page
+- `browser:click` — Click element
+- `browser:fill` — Fill form field
+- `browser:evaluate` — Run JavaScript
+- `browser:network` — Inspect network requests
+
+**Graceful fallback:** When MCP is not available, skills should degrade to Bash (curl, screenshots via CLI) rather than fail.
+
+---
+
+## Layer 2: agent-browser CLI (Vercel)
+
+### Essential Workflow
+
+```bash
+# 1. Navigate
+agent-browser open "https://example.com"
+
+# 2. Snapshot (generates @e1, @e2 element references)
+agent-browser snapshot -i
+
+# 3. Interact using references
+agent-browser click @e5
+agent-browser fill @e3 "search query"
+
+# 4. Re-snapshot after DOM changes
+agent-browser snapshot -i
+```
+
+### Key Commands
+
+| Category    | Commands                                                      |
+| ----------- | ------------------------------------------------------------- |
+| Navigation  | `open`, `close`, `navigate`, `goto`                           |
+| Inspection  | `snapshot -i`, `screenshot`, `pdf`, `get`, `eval`             |
+| Interaction | `click`, `fill`, `type`, `select`, `check`, `press`, `scroll` |
+| Waiting     | `wait` (element, network, URL, text, JS condition, ms)        |
+| State       | `state save/load/list`, session names, auth vault             |
+| Capture     | `screenshot --annotated`, `screenshot --full-page`, `pdf`     |
+| Comparison  | `diff snapshot`, `diff screenshot`, `diff url`                |
+
+### Session Management
+
+```bash
+# Named sessions for parallel work
+agent-browser open "https://site-a.com" --session "task-a"
+agent-browser open "https://site-b.com" --session "task-b"
+
+# Persistent profiles (cookies survive restarts)
+agent-browser open "https://app.com" --profile "my-account"
+
+# State save/restore
+agent-browser state save "logged-in-state"
+agent-browser state load "logged-in-state"
+```
+
+### Authentication Patterns
+
+```bash
+# Connect to existing Chrome (user already logged in)
+agent-browser open "https://app.com" --connect-to-existing
+
+# Encrypted state persistence (generate key with: openssl rand -hex 32)
+AGENT_BROWSER_ENCRYPTION_KEY=$(openssl rand -hex 32) agent-browser state save "auth-state"
+```
+
+### Reference Files
+
+- [references/agent-browser-skill.md](references/agent-browser-skill.md) — Complete 632-line SKILL reference
+- [references/commands.md](references/commands.md) — Full command reference (266 lines)
+- [references/snapshot-refs.md](references/snapshot-refs.md) — Element reference lifecycle
+- [references/session-management.md](references/session-management.md) — Concurrent session patterns
+- [references/authentication.md](references/authentication.md) — Login flows, OAuth, 2FA strategies
+- [references/video-recording.md](references/video-recording.md) — Session recording workflows
+- [references/profiling.md](references/profiling.md) — Performance analysis
+- [references/proxy-support.md](references/proxy-support.md) — Proxy configuration
+
+### Templates
+
+- [templates/form-automation.sh](templates/form-automation.sh) — Form fill workflow
+- [templates/authenticated-session.sh](templates/authenticated-session.sh) — Login + persist session
+- [templates/capture-workflow.sh](templates/capture-workflow.sh) — Screenshot/PDF capture
+
+---
+
+## Layer 3: Playwright Python (Production)
+
+### An Internal ChromeAutomation Engine
+
+Production-proven pattern for persistent Chrome automation with anti-detection:
+
+```python
+# Core architecture (from chrome_automation.py)
+class ChromeAutomation:
+    """Manages Chrome lifecycle with Playwright async API."""
+
+    # Persistent profiles — cookies/sessions survive restarts
+    profile_dir = "~/.<app>/chrome-profiles/{profile_name}"
+
+    # Anti-detection args
+    chrome_args = [
+        "--disable-blink-features=AutomationControlled",
+        "--no-sandbox",
+        "--disable-infobars",
+        "--disable-dev-shm-usage",
+    ]
+
+    # Features:
+    # - Multi-tab support with named pages
+    # - Health checks and auto-recovery
+    # - Human-like interaction (random delays, mouse offsets)
+    # - Network video monitoring (captures MP4/WebM URLs)
+    # - Cookie/session persistence
+    # - File downloads with session cookies
+    # - Headless or headed mode
+```
+
+### Production Automators (internal Chrome-automation engine)
+
+Three standalone automator classes, each with independent rate limiting:
+
+| Automator            | Platform           | Daily Limit           | Rate Delay | Output                                              |
+| -------------------- | ------------------ | --------------------- | ---------- | --------------------------------------------------- |
+| `ImageGenAutomator`  | freepik.com/pikaso | ~unlimited (Premium+) | 5-10s      | Images (Flux, Mystic 2.5) + Videos (MiniMax, Kling) |
+| `ChatImageAutomator` | gemini.google.com  | 300                   | 20-40s     | Images via chat                                     |
+| `VideoGenAutomator`  | meta.ai            | 500                   | 36-60s     | Video clips (8-20s)                                 |
+
+Each implements its own `_daily_limit` and `_enforce_rate_limit()` — there is no shared base class.
+
+### Production Endpoints
+
+```
+POST /platforms/start   — Start Chrome + platform sessions
+POST /platforms/stop    — Stop Chrome + all sessions
+```
+
+### Anti-Detection Checklist (Production Mandatory)
+
+1. Disable `AutomationControlled` Blink feature
+2. Random delays between actions (never fixed intervals)
+3. Mouse offset randomization on clicks
+4. Set locale/timezone (`pt-BR`, `America/Sao_Paulo`)
+5. Fixed viewport `1920x1080` (consistent fingerprint)
+6. Persistent profiles (don't start fresh each time)
+7. Respect rate limits and daily quotas per platform
+8. Handle CAPTCHAs via browser-takeover (human handoff)
+9. Use `ignore_https_errors=True` for certificate edge cases
+
+### Known Bugs & Gotchas (Production-Verified)
+
+> These are real bugs found in a production ChromeAutomation engine. Any agent working on browser automation MUST be aware of these.
+
+**CRITICAL — Auto-exit / Crash issues:**
+
+1. **Stale page references after restart** — When `is_page_healthy()` triggers browser restart, old page objects in `self._pages` become dead. Next `get_page()` returns a closed page. **Fix:** Invalidate all page references on restart.
+
+2. **Missing explicit timeouts on `page.goto()`** — Many navigation calls use Playwright's default 30s timeout, which is too short for heavy pages on Windows. **Fix:** Always pass `timeout=45000` to `page.goto()`.
+
+3. **Race condition in batch restart** — Browser restarts mid-batch with 3s sleep but no re-connection logic. **Fix:** After restart, re-acquire page references before continuing.
+
+4. **Windows process leaks** — Missing `--disable-gpu` flag causes GPU driver crashes on RTX 2060. Chromium processes become zombies. **Fix:** Add `--disable-gpu` to launch args on Windows.
+
+5. **No circuit breaker on reload** — Failed DOM detection triggers `page.reload()` with no max retry count. Can loop forever. **Fix:** Max 2 retries before failing the operation.
+
+**MAJOR — Reliability issues:**
+
+6. **Network monitor state pollution** — Instance-level `_network_monitor` captures URLs from previous requests. **Fix:** Create per-request monitor instances.
+
+7. **Silent download failures** — Download strategies use `except Exception: pass` with no logging. **Fix:** Log every strategy attempt and outcome.
+
+8. **Expired session cookies** — Persistent profiles with month-old sessions fail silently. **Fix:** Validate session before batch start.
+
+9. **Playwright version not pinned** — `requirements.txt` doesn't pin playwright version. Different versions have different Chromium paths. **Fix:** Pin `playwright==1.49.0` (or current version).
+
+10. **`launch_persistent_context` gotcha** — The persistent context IS the browser context (not a separate object). Methods expecting `BrowserContext` get back a `Browser`-like object. Documented nowhere.
+
+---
+
+## Browser Takeover Protocol
+
+**When agents encounter sensitive operations, they MUST NOT automate them.** Instead, use the browser-takeover protocol:
+
+```
+[ASK: BROWSER_TAKEOVER — {url} — Please complete {action} manually]
+```
+
+### Trigger Conditions (Always Hand Off)
+
+| Condition              | Why                                         |
+| ---------------------- | ------------------------------------------- |
+| Login / authentication | Credentials must not be in agent context    |
+| Payment / financial    | Legal liability, PCI compliance             |
+| 2FA / MFA              | Security codes are time-sensitive, personal |
+| CAPTCHA                | By design requires human verification       |
+| Personal data entry    | Privacy, GDPR, consent                      |
+| OAuth consent screens  | User must explicitly grant permissions      |
+
+### How It Works
+
+1. Agent detects sensitive operation needed
+2. Agent opens browser to the correct page/URL
+3. Agent prints `[ASK: BROWSER_TAKEOVER — ...]` message
+4. User completes the action manually
+5. Agent resumes automation after user confirms completion
+
+See the `browser-takeover` general skill for the full protocol.
+
+---
+
+## Environment Variables
+
+| Variable                        | Purpose                     | Layer         |
+| ------------------------------- | --------------------------- | ------------- |
+| `AGENT_BROWSER_ENCRYPTION_KEY`  | State file encryption       | agent-browser |
+| `AGENT_BROWSER_DEFAULT_TIMEOUT` | Timeout (ms)                | agent-browser |
+| `AGENT_BROWSER_IDLE_TIMEOUT_MS` | Auto-shutdown delay         | agent-browser |
+| `AGENT_BROWSER_ALLOWED_DOMAINS` | Navigation allowlist        | agent-browser |
+| `AGENT_BROWSER_ACTION_POLICY`   | Operation restrictions      | agent-browser |
+| `AGENT_BROWSER_HEADED`          | Show browser window         | agent-browser |
+| `AGENT_BROWSER_ENGINE`          | `chrome` or `lightpanda`    | agent-browser |
+| `PLAYWRIGHT_BROWSERS_PATH`      | Custom browser install path | Playwright    |
+
+---
+
+## Critical Rules
+
+1. **NEVER automate login, payment, 2FA, CAPTCHA** — Always use browser-takeover protocol
+2. **Rate limiting is mandatory** — Every platform automator must have daily quotas and random delays
+3. **Anti-detection for production** — Disable AutomationControlled, random delays, persistent profiles
+4. **Prefer MCP when available** — Falls back to agent-browser CLI, then Playwright Python
+5. **Snapshot before interact** — Always `snapshot -i` to get fresh element references before clicking/filling
+6. **Re-snapshot after DOM changes** — Element references (`@e1`, `@e2`) are invalidated by page mutations
+7. **Persistent profiles for auth** — Never re-authenticate when profile cookies are available
+8. **Headless by default, headed for debugging** — Only use headed mode during development