testchimp-runner-core 0.0.33 → 0.0.35

package/plandocs/VISUAL_AGENT_EVOLUTION_PLAN.md

@@ -0,0 +1,396 @@
+# Runner-Core Visual Agent Evolution - Complete Implementation Plan
+
+## Overview
+
+Two-phase pragmatic evolution without major architecture overhaul:
+- **Phase 1 (Week 1-2):** Percentage coordinates + Free-form notes
+- **Phase 2 (Week 3-5):** Numbered element system with three-tier fallback
+
+---
+
+## PHASE 1: Tactical Improvements
+
+### 1A: Free-Form "Note to Future Self"
+
+**Why:** Agent needs tactical memory between iterations of the SAME step.
+
+**Type:**
+```typescript
+interface NoteToFutureSelf {
+  fromIteration: number;
+  content: string;  // FREE-FORM - agent writes whatever it wants
+}
+```
+
+**Examples:**
+- "Tried #sidebar-toggle, failed. Will try SVG child next."
+- "Plan: Hover over menu first to reveal dropdown, then click Settings."
+- "Cookie banner blocking. Next: dismiss it, then retry main action."
+
+**vs Current Learnings:**
+- Learnings = App-wide patterns ("App uses getByRole")
+- Note to self = Iteration-specific tactics ("Just tried X, will try Y next")
+
+**Keep BOTH!**
+
+### 1B: Percentage-Based Coordinates
+
+**LLM outputs percentages:**
+```json
+{
+  "coordinateAction": {
+    "action": "click|fill|drag|hover|scroll",
+    "xPercent": 15.75,  // 2 decimal precision for accuracy
+    "yPercent": 8.50,
+    
+    // For drag:
+    "toXPercent": 45.25,
+    "toYPercent": 8.50,
+    
+    // For fill:
+    "value": "alice@example.com"
+  }
+}
+```
+
+**Code converts to pixels:**
+```typescript
+const viewport = await page.evaluate(() => ({width: window.innerWidth, height: window.innerHeight}));
+const x = Math.round((xPercent / 100) * viewport.width);
+const y = Math.round((yPercent / 100) * viewport.height);
+```
+
+### 1C: Code-Controlled Fallback
+
+**Tier 1 (iteration 1-3):** Normal Playwright selectors
+**Tier 2 (iteration 4+):** Percentage coordinates (auto-triggered after 3 failures)
+
+```typescript
+// In callAgent():
+if (consecutiveFailures >= 3) {  // Phase 1: Only 2 tiers
+  // Auto-use coordinate-specific system prompt
+  // LLM outputs percentages
+  // Phase 2 will add tier 2 (indexed elements) between selectors and coordinates
+}
+```
+
+**Files:**
+- `utils/coordinate-converter.ts` (NEW)
+- `orchestrator/orchestrator-agent.ts` (UPDATE)
+- `orchestrator/types.ts` (UPDATE - add CoordinateAction, NoteToFutureSelf)
+
+---
+
+## PHASE 2: Numbered Element System
+
+### Architecture
+
+```
+Three-Tier Fallback:
+
+Tier 1 (iteration 1 ONLY): Playwright Selector Mode - ONE SHOT
+  ├─ Agent generates: await page.getByRole('button', {name: 'Login'}).click()
+  ├─ Direct execution
+  └─ 70% of tasks finish here (simple/medium complexity)
+
+Tier 2 (iterations 2-3): Index Command Mode - TWO ATTEMPTS
+  ├─ Inject numbered markers [1], [2], [3] → screenshot
+  ├─ Agent outputs: CLICK[3], FILL[5, "alice@example.com"]
+  ├─ Execution: Use data-testchimp-el="[3]" (reliable targeting)
+  ├─ Script output: Translate to NATIVE selector (getByRole, #id, etc.)
+  └─ 25% of tasks finish here (complex UIs, icons, shadow DOM)
+
+Tier 3 (iterations 4+): Percentage Coordinate Mode - LAST RESORT
+  ├─ Agent outputs: {xPercent: 15.755, yPercent: 8.500}
+  ├─ CoordinateConverter: % → pixels
+  ├─ Execute: mouse.click(x, y)
+  └─ <5% of tasks need this (extreme edge cases)
+```
+
+### 2.1: Reusable Utility: ElementDetector
+
+**File:** `runner-core/src/utils/element-detector.ts`
+
+**Purpose:** Detect ALL interactive elements with z-index and occlusion awareness.
+
+**Key Features:**
+- Comprehensive element queries (buttons, links, inputs, SVGs, clickable divs)
+- Z-index calculation via `getComputedStyle()`
+- Occlusion detection via `elementFromPoint()` at center
+- Context tags: `[header|nav|sidebar|main|modal]`
+- Spatial tags: `[top|bottom|left|right|center]`
+- Center position as percentage (2 decimal precision)
+
+**Output:**
+```typescript
+interface DetectedElement {
+  index: number;                    // [1], [2], [3]
+  tag: string;
+  text: string;
+  role: string;
+  ariaLabel: string;
+  bbox: {x, y, width, height};
+  centerPercent: {x: 15.75, y: 8.50};  // As percentage
+  context: string[];                // ['header', 'nav', 'top']
+  zIndex: number;
+  isVisible: boolean;               // Not occluded by higher z-index
+  selectors: {
+    dataAttribute: string;          // [data-testchimp-el="[3]"]
+    semantic: string[];             // [getByRole(...), getByLabel(...)]
+    cssId: string | null;
+    cssClass: string | null;
+  };
+}
+```
+
+**Implementation Highlights:**
+- Queries: `button`, `a[href]`, `input`, `svg`, `[onclick]`, `[role=...]`, `[data-testid]`, clickable divs/spans
+- Z-index check: `elementFromPoint(centerX, centerY)` must return element or child
+- Filter: Only return `isVisible: true` elements
+
+### 2.2: Reusable Utility: SelectorResolver
+
+**File:** `runner-core/src/utils/selector-resolver.ts`
+
+**Purpose:** Given element index, return most reliable Playwright selector FOR GENERATED SCRIPTS.
+
+**CRITICAL DISTINCTION:**
+
+A. **During Execution (Internal):**
+   - Agent can click using `data-testchimp-el="[N]"` (we inject it temporarily)
+   - This ensures agent clicks the exact right element
+
+B. **For Generated Script (Output):**
+   - Must use NATIVE selectors that work without our attributes
+   - Script will run on real application without data-testchimp-el
+
+**Selector Resolution for Script Output (in order):**
+1. Semantic selectors - `getByRole()`, `getByLabel()` (BEST - maintainable)
+2. CSS ID - `#element-id` (GOOD - stable)
+3. CSS class - `.button-primary` (scoped to context if ambiguous)
+4. Contextual selector - `header .menu-toggle svg` (LAST RESORT)
+
+**For each selector, validate:**
+- Element exists on page
+- Not covered by higher z-index (using elementFromPoint)
+- Position matches expected (±5% tolerance)
+
+**Method:**
+```typescript
+static async resolveIndexToSelector(
+  index: number,
+  elements: DetectedElement[],
+  page: Page
+): Promise<{selector: string, strategy: string}>
+```
+
+**Validation:**
+```typescript
+private static async validateInteractable(
+  page: Page,
+  selector: string,
+  expectedCenterPercent: {x, y}
+): Promise<boolean> {
+  // 1. Element exists
+  // 2. Non-zero dimensions
+  // 3. elementFromPoint(center) === element (z-index check)
+  // 4. Position within ±5% tolerance
+}
+```
+
+### 2.3: Reusable Utility: VisualMarkerInjector
+
+**File:** `runner-core/src/utils/visual-marker-injector.ts`
+
+**Purpose:** Inject visual numbered labels on page.
+
+**Methods:**
+```typescript
+static async injectNumberedMarkers(page): Promise<DetectedElement[]>
+static async removeMarkers(page): Promise<void>
+static async captureMarkedScreenshot(page, elements): Promise<string>
+```
+
+**Visual Markers:**
+- Red gradient background `[1]`, `[2]`, `[3]`
+- Positioned at top-left of element
+- z-index: 999999 (always visible)
+- Inject `data-testchimp-el` attribute on actual element (TEMPORARY - for agent execution only)
+  * Used internally to ensure agent clicks correct element
+  * Removed after step completion
+  * NEVER appears in generated script output
+
+### 2.4: Reusable Utility: IndexCommandTranslator
+
+**File:** `runner-core/src/utils/index-command-translator.ts`
+
+**Purpose:** Translate index commands to Playwright commands with NATIVE selectors (for script generation).
+
+**Input:**
+```typescript
+{ action: "CLICK", index: 3 }
+{ action: "FILL", index: 5, value: "alice@example.com" }
+```
+
+**Output (MUST use native selectors):**
+```typescript
+"await page.getByRole('button', {name: 'Menu'}).click();"
+"await page.locator('#username').fill('alice@example.com');"
+// OR
+"await page.locator('#sidebar-toggle svg').click();"
+```
+
+**NOT this (won't work in generated script):**
+```typescript
+❌ "await page.locator('[data-testchimp-el=\"[3]\"]').click();"
+❌ "await page.locator('[data-testchimp-el=\"[5]\"]').fill('alice@example.com');"
+```
+
+**Process:**
+1. **During execution:** Click element using `data-testchimp-el="[index]"` (reliable)
+2. **For script output:** Use SelectorResolver to get NATIVE selector (semantic/id/class)
+3. Generate Playwright command with native selector
+4. Return command string that works on real application
+
+**Critical Distinction:**
+- `data-testchimp-el` = Internal execution helper (temporary)
+- Script output = Native selectors (permanent, works standalone)
+
+### 2.5: Integration - Three-Tier System (Optimized Escalation)
+
+**File:** `orchestrator/orchestrator-agent.ts`
+
+**Optimized Strategy:** Escalate quickly to avoid wasting time on difficult tasks
+
+**Mode Determination:**
+```typescript
+let tier: 1 | 2 | 3;
+
+// Tier 1: Try normal selectors ONCE (iteration 1)
+// Tier 2: Use indexed elements TWICE (iterations 2-3) 
+// Tier 3: Use percentage coords (iterations 4+)
+
+if (iteration >= 4) {
+  tier = 3;  // Coordinate mode
+} else if (iteration >= 2) {
+  tier = 2;  // Index command mode
+} else {
+  tier = 1;  // Normal Playwright selector mode
+}
+```
+
+**Rationale:**
+- Simple tasks: Succeed in Tier 1 (iteration 1) - fast!
+- Medium tasks: Tier 2 gives 2 attempts with reliable index system
+- Hard tasks: Tier 3 coordinates as absolute fallback
+- No wasted iterations on difficult element detection
+
+**Tier Preparation:**
+```typescript
+if (tier === 2) await this.prepareIndexMode(context, page);
+if (tier === 3) await this.prepareCoordinateMode(context, page);
+```
+
+**System Prompt Selection:**
+```typescript
+const systemPrompt = 
+  tier === 3 ? this.buildCoordinateSystemPrompt() :
+  tier === 2 ? this.buildIndexSystemPrompt() :
+  this.buildSystemPrompt();
+```
+
+**Execution Flow:**
+```typescript
+// Tier 1 (iteration 1):
+if (decision.commands) {
+  // Execute normal Playwright command
+  // If fails → iteration++, move to Tier 2
+}
+
+// Tier 2 (iterations 2-3):
+if (decision.indexCommand) {
+  // Step A: Click using data-testchimp-el="[N]" (execution)
+  await page.locator('[data-testchimp-el="[3]"]').click();
+  
+  // Step B: Resolve to native selector (script generation)
+  const nativeSelector = await SelectorResolver.resolve(3, elements);
+  // → Returns: "getByRole('button', {name: 'Menu'})"
+  
+  // Step C: Add to generated script
+  commandsExecuted.push(`await page.${nativeSelector}.click();`);
+  
+  // If fails after 2 attempts → iteration++, move to Tier 3
+}
+
+// Tier 3 (iterations 4+):
+if (decision.coordinateAction) {
+  // Convert % to pixels and execute
+  // Add coordinate commands to script (acceptable for edge cases)
+}
+```
+
+---
+
+## Utilities Summary
+
+All utilities are **stateless and reusable**:
+
+| Utility | Purpose | Reusable For |
+|---------|---------|--------------|
+| ElementDetector | Find interactive elements | Accessibility audits, page analysis |
+| SelectorResolver | Index → selector with validation | Any numbered system |
+| VisualMarkerInjector | Add visual labels | Manual testing, debugging |
+| IndexCommandTranslator | Index command → Playwright | Any index-based automation |
+| CoordinateConverter | Percentage → pixels | Any coordinate system |
+
+---
+
+## Implementation Timeline
+
+### Week 1: Phase 1 Core
+- [ ] NoteToFutureSelf type and tracking
+- [ ] CoordinateAction with percentages
+- [ ] CoordinateConverter utility
+- [ ] Coordinate mode switching (tier 3)
+
+### Week 2: Phase 1 Testing
+- [ ] Test note-to-self on 10 scenarios
+- [ ] Test percentage coordinates at multiple viewport sizes
+- [ ] Verify all coordinate actions (click, fill, drag, scroll, hover)
+
+### Week 3: Phase 2 Utilities
+- [ ] ElementDetector with z-index awareness
+- [ ] SelectorResolver with occlusion validation
+- [ ] Test utilities standalone on complex pages
+
+### Week 4: Phase 2 Integration
+- [ ] VisualMarkerInjector
+- [ ] IndexCommandTranslator (TWO-STAGE: execution via data-attr, script via native selector)
+- [ ] Index mode (tier 2) integration with iteration-based switching
+- [ ] Optimized escalation: iteration 1 → tier 1, iteration 2-3 → tier 2, iteration 4+ → tier 3
+- [ ] Test PeopleHR with tier 2 (should succeed in iteration 2-3)
+
+### Week 5: Phase 2 Testing
+- [ ] Three-tier end-to-end testing
+- [ ] Measure tier distribution (target: 70/25/5)
+- [ ] A/B test vs current implementation
+- [ ] Performance optimization
+
+## Success Metrics
+
+**Phase 1:**
+- 20-30% reduction in average iterations per step
+- Note-to-self prevents 40%+ of repeated selector failures
+- Coordinates used in < 5% of scenarios
+
+**Phase 2:**
+- 70% scenarios complete in Tier 1 (iteration 1) - simple cases
+- 25% scenarios use Tier 2 (iterations 2-3) - complex UIs with icons/shadows
+- < 5% scenarios escalate to Tier 3 (iterations 4+) - impossible selector cases
+- **PeopleHR hamburger menu:** Succeeds in Tier 2 iteration 2 with CLICK[N]
+- **Average iterations per step:** Should decrease from ~4 to ~1.5
+
+## Total Effort: 4-5 weeks
+
+**Ready to implement?**
+

package/plandocs/WHATS_NEW_v0.0.33.md

@@ -0,0 +1,183 @@
+# What's New in Runner-Core v0.0.33
+
+## Phase 1: Tactical Improvements - COMPLETE ✅
+
+---
+
+## 1. 📝 Note to Future Self (Cross-Step Memory)
+
+**The agent can now leave notes that persist across the entire scenario journey.**
+
+### How it works:
+```typescript
+// Step 1 - Login
+Agent: "Cookie modal appears after 2s. Dismiss it before interacting."
+→ Stored in memory.latestNote
+
+// Step 2 - Navigate to Dashboard
+Agent reads note from Step 1
+Agent: "Waiting 2s for cookie modal..."
+→ Dismisses modal proactively
+```
+
+### Scope:
+- ✅ Across iterations (within same step)
+- ✅ Across steps (entire scenario)
+- ✅ Free-form text (agent decides what's important)
+
+### Example notes:
+- **Tactical:** "Tried #menu, failed. Try SVG child next."
+- **Strategic:** "This app uses shadow DOM. Prefer CSS selectors over getByRole."
+- **Behavioral:** "Modals load after 2s delay. Wait before clicking."
+
+---
+
+## 2. 🎯 Percentage-Based Coordinate Fallback
+
+**When selectors fail, use visual positioning as last resort.**
+
+### Precision:
+- 3 decimal places (e.g., 15.755%, 8.500%)
+- ~1 pixel accuracy on most screens
+- Resolution-independent
+
+### Supported Actions:
+- **Click:** `{action: "click", xPercent: 15.755, yPercent: 8.500}`
+- **Fill:** `{action: "fill", xPercent: 30.000, yPercent: 25.000, value: "text"}`
+- **Drag:** `{action: "drag", xPercent: 10.000, yPercent: 50.000, toXPercent: 60.000, toYPercent: 50.000}`
+- **Hover, RightClick, DoubleClick, Scroll**
+
+### Auto-Activation:
+- Triggers after 3 consecutive selector failures
+- Limited to 2 coordinate attempts
+- Then gives up (stuck)
+
+---
+
+## 3. ⚡ Optimized Iteration Budget
+
+**Maximum 5 iterations per step** (down from 8)
+
+```
+Iterations 1-3: Playwright selectors (3 attempts)
+    with note-to-self between each
+    
+Iterations 4-5: Coordinates (2 attempts max)
+    If both fail → stuck
+```
+
+**Why:** Coordinates either work or don't - no point retrying 5+ times.
+
+---
+
+## 4. 🕐 Smart Timeout Handling (Earlier Fix)
+
+**Navigation operations now have appropriate timeouts:**
+- `waitForLoadState()`: 30 seconds (was 5s)
+- `goto()`: 30 seconds
+- Element operations: 5 seconds (unchanged)
+
+**Detects automatically:** Code scans command for navigation keywords.
+
+---
+
+## How Notes Work Across Steps
+
+### Example Scenario:
+
+```
+Step 1: Login
+  Iteration 1: Fill username → Success
+  Iteration 2: Fill password → Success
+  Iteration 3: Click login → Success
+  Agent note: "Login redirects to dashboard. Cookie modal appears after 2s."
+  
+Step 2: Navigate to Settings
+  Reads note from Step 1: "Cookie modal appears after 2s"
+  Iteration 1: 
+    - Wait 2s
+    - Dismiss modal
+    - Click Settings
+    → Success in 1 iteration! (note prevented wasted attempts)
+```
+
+**Benefit:** Agent builds up knowledge about the application and uses it in future steps.
+
+---
+
+## Comparison: Before vs After
+
+| Aspect | Before (v0.0.32) | After (v0.0.33) |
+|--------|------------------|------------------|
+| Iteration memory | None | Note to self (cross-step) |
+| Selector fails | Give up or loop | Coordinate fallback |
+| Max iterations | 8 per step | 5 per step |
+| Timeout handling | 5s for all | 30s for navigation |
+| Coordinate support | None | Full (click, fill, drag, etc.) |
+| Average iterations | ~4 per step | ~2.5 per step (estimated) |
+
+---
+
+## Testing Recommendations
+
+### Test 1: Note Continuity
+Create a scenario with repeated patterns:
+```
+- Login
+- Go to page A → encounter modal
+- Go to page B → should handle modal proactively
+```
+
+**Expected:** Step 2 learns from Step 1's note.
+
+### Test 2: Coordinate Fallback  
+Run PeopleHR scenario:
+```
+- Click hamburger menu (SVG icon)
+```
+
+**Expected:**
+- Iterations 1-3: Try selectors (may fail)
+- Iteration 4: Coordinates → succeeds
+- Generated script: `await page.mouse.click(x, y);`
+
+### Test 3: Timeout Fix
+Any scenario with:
+```
+- await page.waitForLoadState('networkidle');
+```
+
+**Expected:** No more 5s timeout errors.
+
+---
+
+## Migration
+
+**No code changes needed!** Existing code works as-is with improvements.
+
+**New response fields** (optional):
+- `noteToFutureSelf`: string (agent can optionally include)
+- `coordinateAction`: object (only when coordinate mode active)
+
+---
+
+## What's Next: Phase 2
+
+Phase 2 will add numbered element system for even better reliability:
+- Iteration 1: Playwright selector (1 attempt)
+- Iterations 2-3: Index commands CLICK[3] (2 attempts)
+- Iterations 4-5: Coordinates (2 attempts)
+
+**Target:** ~1.5 average iterations per step
+
+---
+
+## Status
+
+✅ **Built and Ready**  
+📦 **Version:** v0.0.33  
+🧪 **Status:** Ready for testing  
+📊 **Expected Impact:** 30-40% reduction in iterations  
+
+**Test now to validate improvements before Phase 2!**
+