linkedin-automation-cli 1.0.0

package/docs/superpowers/specs/2026-03-14-page-agent-design.md

@@ -0,0 +1,385 @@
+# Page Agent Integration Design
+
+**Date:** 2026-03-14
+**Status:** Draft
+**Scope:** Replace selector-based automation with DOM-text + LLM agent approach
+
+---
+
+## Overview
+
+Replace the current CSS selector-based LinkedIn automation with a DOM-text extraction and LLM-powered agent system, inspired by Alibaba's Page Agent. This approach extracts semantic text representations of the page and uses Claude API (via Dashscope) to make decisions about actions.
+
+---
+
+## Current Architecture (v0.3.0)
+
+```
+User Command → CLI → SelectorEngine → Playwright → CSS Selectors → LinkedIn
+                    ↓
+              (Multi-layer: a11y → text → pattern → legacy)
+```
+
+**Problems:**
+- Brittle CSS selectors break when LinkedIn updates UI
+- Selector learning system adds complexity
+- Hard-coded selector chains are maintenance burden
+- No natural language understanding
+
+---
+
+## New Architecture
+
+```
+User Command → CLI → PageAgent → DOM Extractor → Claude API → Action Plan → Playwright → LinkedIn
+              ↓                                          ↓
+         (Natural language)                       (Structured actions)
+```
+
+---
+
+## Components
+
+### 1. DOM Extractor (`src/agent/dom-extractor.ts`)
+
+Extracts semantic text representation of the LinkedIn page using Playwright.
+
+**Output format:**
+```typescript
+interface DOMRepresentation {
+  url: string;
+  title: string;
+  elements: DOMElement[];
+  textContent: string;
+  interactiveElements: InteractiveElement[];
+}
+
+interface DOMElement {
+  id: string;
+  tag: string;
+  role?: string;
+  ariaLabel?: string;
+  text: string;
+  bbox?: { x: number; y: number; width: number; height: number };
+}
+
+interface InteractiveElement {
+  id: string;
+  type: 'button' | 'link' | 'input' | 'textarea' | 'select';
+  text: string;
+  purpose: string; // inferred from context
+}
+```
+
+**Extraction strategy:**
+- Use `page.evaluate()` to extract clean DOM structure
+- Focus on interactive elements (buttons, links, inputs)
+- Include ARIA labels, roles, and visible text
+- Remove script/style content, keep semantic structure
+
+### 2. Claude API Client (`src/agent/claude-client.ts`)
+
+Client for Dashscope Claude API endpoint.
+
+**Configuration:**
+- Base URL: `https://coding.dashscope.aliyuncs.com/apps/anthropic/v1`
+- Model: `qwen3.5-plus`
+- API Key: From `.env` file (`DASHSCOPE_API_KEY`)
+
+**Interface:**
+```typescript
+interface ClaudeClient {
+  generateActionPlan(
+    dom: DOMRepresentation,
+    goal: string,
+    context?: ActionContext
+  ): Promise<ActionPlan>;
+}
+
+interface ActionPlan {
+  actions: Action[];
+  reasoning: string;
+  expectedOutcome: string;
+}
+
+type Action =
+  | { type: 'click'; elementId: string; description: string }
+  | { type: 'type'; elementId: string; text: string; description: string }
+  | { type: 'wait'; durationMs: number; description: string }
+  | { type: 'navigate'; url: string; description: string }
+  | { type: 'verify'; condition: string; description: string };
+```
+
+### 3. Action Executor (`src/agent/action-executor.ts`)
+
+Executes action plans using Playwright.
+
+**Responsibilities:**
+- Map element IDs to actual DOM elements
+- Execute clicks, typing, navigation
+- Handle errors and retries
+- Verify outcomes
+
+### 4. Page Agent (`src/agent/page-agent.ts`)
+
+Main orchestrator combining all components.
+
+**Interface:**
+```typescript
+interface PageAgent {
+  // High-level task execution
+  execute(task: Task): Promise<TaskResult>;
+
+  // Specific tasks
+  connect(profileUrl: string, note?: string): Promise<ConnectionResult>;
+  sendMessage(profileUrl: string, message: string): Promise<MessageResult>;
+}
+```
+
+---
+
+## Prompt Design
+
+### System Prompt
+
+```
+You are a web automation agent that controls LinkedIn through a browser.
+You receive a DOM representation of the current page and must decide the next action(s) to achieve the user's goal.
+
+Rules:
+1. Respond ONLY with a JSON action plan
+2. Each action must reference an element by its ID from the DOM
+3. Include brief reasoning for your decision
+4. If the goal is already achieved, return empty actions with success status
+5. If stuck or error, explain why and suggest recovery
+
+Available actions:
+- click: Click an element
+- type: Type text into an input
+- wait: Wait for page to stabilize
+- navigate: Go to a URL
+- verify: Check if a condition is met
+
+Response format:
+{
+  "reasoning": "string",
+  "expectedOutcome": "string",
+  "actions": [
+    { "type": "click", "elementId": "elem-123", "description": "Click Connect button" }
+  ],
+  "status": "in_progress" | "completed" | "error"
+}
+```
+
+### User Prompt Template
+
+```
+Goal: {{userGoal}}
+
+Current page DOM:
+```json
+{{domRepresentation}}
+```
+
+{{#if previousActions}}
+Previous actions taken:
+{{#each previousActions}}
+- {{type}}: {{description}} ({{status}})
+{{/each}}
+{{/if}}
+
+What action(s) should be taken next?
+```
+
+---
+
+## DOM Extraction Strategy
+
+### Approach: Accessibility Tree + Interactive Elements
+
+Extract elements in priority order:
+
+1. **Buttons** - `button` elements, `[role="button"]`, clickables
+2. **Links** - `a[href]` elements
+3. **Form inputs** - `input`, `textarea`, `select`
+4. **Text content** - Headings, labels, status messages
+
+### Element ID Assignment
+
+Assign stable IDs based on element characteristics:
+- Use `data-testid` if available
+- Fallback to `[tag]-[aria-label]-[index]` hash
+- Include bounding box for click coordinates
+
+### Example Extracted DOM
+
+```json
+{
+  "url": "https://www.linkedin.com/in/alice",
+  "title": "Alice Smith | LinkedIn",
+  "elements": [
+    {
+      "id": "btn-connect-1",
+      "tag": "button",
+      "role": "button",
+      "ariaLabel": "Connect",
+      "text": "Connect",
+      "bbox": { "x": 800, "y": 200, "width": 100, "height": 40 }
+    },
+    {
+      "id": "btn-message-1",
+      "tag": "button",
+      "role": "button",
+      "ariaLabel": "Message",
+      "text": "Message",
+      "bbox": { "x": 920, "y": 200, "width": 100, "height": 40 }
+    },
+    {
+      "id": "input-note-1",
+      "tag": "textarea",
+      "role": "textbox",
+      "ariaLabel": "Add a note",
+      "text": "",
+      "bbox": { "x": 500, "y": 400, "width": 400, "height": 100 }
+    }
+  ],
+  "status": {
+    "connectionState": "not_connected",
+    "profileName": "Alice Smith",
+    "profileTitle": "Software Engineer at TechCorp"
+  }
+}
+```
+
+---
+
+## Action Execution Flow
+
+### Connect Flow
+
+```
+1. Navigate to profile URL
+2. Extract DOM
+3. Send to Claude: "Send connection request to this profile"
+4. Receive action plan:
+   - Click Connect button
+   - Wait for modal
+   - Type note (if provided)
+   - Click Send
+5. Execute actions one by one
+6. Verify outcome (check for Pending status)
+```
+
+### Message Flow
+
+```
+1. Navigate to profile URL
+2. Extract DOM
+3. Send to Claude: "Send message to this person"
+4. Receive action plan:
+   - Click Message button
+   - Wait for composer
+   - Type message
+   - Click Send
+5. Execute actions
+6. Verify outcome
+```
+
+---
+
+## Error Handling
+
+### Recovery Strategies
+
+1. **Element not found** - Re-extract DOM, retry with updated IDs
+2. **Action timeout** - Wait longer, check if page changed
+3. **Unexpected state** - Re-plan with new DOM
+4. **LLM error** - Retry with exponential backoff
+
+### Safety Checks
+
+- Validate URLs are LinkedIn profiles before navigation
+- Rate limiting via existing limiter
+- Confirmation prompts for destructive actions
+
+---
+
+## File Structure
+
+```
+src/
+├── agent/
+│   ├── index.ts           # Public API exports
+│   ├── page-agent.ts      # Main orchestrator
+│   ├── dom-extractor.ts   # DOM to text extraction
+│   ├── claude-client.ts   # Claude API client
+│   ├── action-executor.ts # Playwright action execution
+│   ├── prompts.ts         # LLM prompts
+│   └── types.ts           # Agent types
+├── cli/
+│   └── agent-command.ts   # New CLI command: `linkedin agent`
+└── linkedin/
+    └── (existing files - keep for fallback)
+```
+
+---
+
+## Migration Plan
+
+### Phase 1: New Agent Commands (This PR)
+- Create `src/agent/` module
+- Implement DOM extractor
+- Implement Claude client
+- Add `linkedin agent connect <url>` command
+- Add `linkedin agent message <url> <text>` command
+- Keep existing commands unchanged
+
+### Phase 2: Replace Existing Commands (Future)
+- Migrate `linkedin connect` to use agent
+- Migrate `linkedin message` to use agent
+- Deprecate selector engine
+
+---
+
+## Environment Configuration
+
+Add to `.env`:
+
+```bash
+# Dashscope Claude API
+DASHSCOPE_API_KEY=your_key_here
+DASHSCOPE_BASE_URL=https://coding.dashscope.aliyuncs.com/apps/anthropic/v1
+DASHSCOPE_MODEL=qwen3.5-plus
+```
+
+---
+
+## Testing Strategy
+
+1. **Unit tests** - DOM extractor with mock pages
+2. **Integration tests** - Agent with mock Claude responses
+3. **E2E tests** - Full flow with test LinkedIn accounts
+
+---
+
+## Trade-offs
+
+### Advantages
+- Resilient to UI changes (text-based, not selectors)
+- Natural language understanding
+- Easier maintenance
+- Self-documenting actions (LLM reasoning)
+
+### Disadvantages
+- Requires API calls (latency, cost)
+- Depends on LLM availability
+- More complex debugging
+
+---
+
+## Success Criteria
+
+1. Connect and message commands work reliably on standard LinkedIn profiles
+2. No hard-coded selectors in new agent code
+3. Clear action logging for debugging
+4. Graceful fallback if LLM is unavailable