testchimp-runner-core 0.0.33 → 0.0.35

package/plandocs/BEFORE_AFTER_VERIFICATION.md

@@ -0,0 +1,148 @@
+# Before/After Screenshot Verification
+
+## Feature: Visual Goal Verification for Coordinate Actions
+
+### Problem Solved:
+When using coordinate-based actions (clicking at x,y%), the agent has no way to know if the click achieved the goal:
+- No element reference to check state
+- No selector feedback
+- Can't verify if expected page loaded or modal opened
+
+This led to:
+- False positives (click succeeded but goal not achieved)
+- Infinite loops (agent keeps clicking, unsure if it worked)
+
+### Solution:
+Automatic before/after screenshot comparison after coordinate clicks.
+
+## How It Works:
+
+### 1. **Automatic Trigger** (No Agent Action Required)
+When agent uses coordinate action:
+```typescript
+Iteration 4: 🎯 Coordinate mode activated
+  Step 1: Capture BEFORE screenshot
+  Step 2: Execute coordinate click (x%, y%)
+  Step 3: Wait 1000ms for UI to settle
+  Step 4: Capture AFTER screenshot
+  Step 5: Call LLM with both images (labeled "BEFORE", "AFTER")
+  Step 6: LLM responds: { goalAchieved: true/false, reasoning: "..." }
+  Step 7a: If TRUE → Mark complete, exit step ✅
+  Step 7b: If FALSE → Continue to next iteration, try different coordinates
+```
+
+### 2. **LLM Prompt for Verification**
+```
+Goal: [Current step goal]
+
+Compare the BEFORE and AFTER screenshots.
+
+Did the action achieve the goal? Respond with JSON:
+{
+  "goalAchieved": boolean,
+  "reasoning": "What changed (or didn't change)",
+  "visibleChanges": ["List of UI changes observed"]
+}
+
+Focus on:
+- Did expected elements appear/disappear?
+- Did page navigate or content change?
+- Visual indicators of success (new panels, forms, highlights)?
+
+Be strict: Only return true if you clearly see the expected change.
+```
+
+### 3. **Multi-Image LLM Interface**
+```typescript
+// NEW: LabeledImage interface
+export interface LabeledImage {
+  label: string;      // "Before", "After", etc.
+  dataUrl: string;    // Base64 data URL
+}
+
+// UPDATED: LLMRequest
+export interface LLMRequest {
+  imageUrl?: string;         // Backward compatible (single image)
+  images?: LabeledImage[];   // NEW - multi-image support
+}
+```
+
+### 4. **Provider Implementation** (scriptservice-llm-provider.ts)
+```typescript
+if (request.images && request.images.length > 0) {
+  for (const img of request.images) {
+    contentParts.push({ type: 'text', text: `\n[${img.label}]:` });
+    contentParts.push({ type: 'image_url', image_url: { url: img.dataUrl } });
+  }
+  // Sends: [BEFORE]: <image1>, [AFTER]: <image2>
+}
+```
+
+## When Verification Happens:
+
+✅ **Always**: After first coordinate action attempt
+❌ **Never**: After selector-based actions (have element state to check)
+⚠️ **Conditional**: Can add for other scenarios where goal verification is unclear
+
+## Cost Considerations:
+
+**Per verification call:**
+- 2 viewport screenshots (~50-100KB each)
+- Vision model (gpt-5-mini): ~$0.001 per call
+- Used only when coordinate mode activates (after 3 selector failures)
+
+**Typical scenario:**
+- Steps 1-10: Regular selectors → No verification cost
+- Step 5 gets stuck → Coordinate mode → 1 verification call → $0.001
+- Overall impact: Minimal, used sparingly
+
+## Example Flow:
+
+**Step 5: "Select Employee Information"**
+```
+Iteration 1: getByText('Employee Information') → Strict mode ❌
+Iteration 2: locator('#collapse-1').getByText('Employee Information') → Click succeeds ✅
+           BUT: Didn't navigate to Employee Information page (false positive)
+           
+Iteration 3: Selector fails again
+Iteration 4: 🎯 Coordinate mode
+  → BEFORE: Homepage with sidebar
+  → Click at (19.3%, 22.9%)
+  → Wait 1s
+  → AFTER: Check screenshot
+  → LLM: "goalAchieved": true, "reasoning": "Employee Information page loaded with form"
+  → ✅ Mark complete, exit
+```
+
+## Backward Compatibility:
+
+✅ **Single image still works:**
+```typescript
+const request = {
+  imageUrl: 'data:image/png;base64,...'  // Old way
+};
+```
+
+✅ **Multi-image NEW:**
+```typescript
+const request = {
+  images: [
+    { label: 'BEFORE', dataUrl: '...' },
+    { label: 'AFTER', dataUrl: '...' }
+  ]
+};
+```
+
+## Files Modified:
+
+1. `runner-core/src/llm-provider.ts` - Added LabeledImage interface and images field
+2. `scriptservice/providers/scriptservice-llm-provider.ts` - Handle multiple images in OpenAI API
+3. `runner-core/src/orchestrator/orchestrator-agent.ts` - Added verifyGoalWithScreenshotComparison method
+4. Automatic trigger after coordinate actions
+
+## Next Steps:
+
+- ✅ Infrastructure ready
+- ⏳ Need to test with real scenario
+- 🔮 Future: Could expose as agent-callable tool if needed
+

package/plandocs/COORDINATE_MODE_DIAGNOSIS.md

@@ -0,0 +1,144 @@
+# Coordinate Mode Diagnosis - Live Test Results
+
+## Test Scenario: PeopleHR Employee Information Flow
+
+### ✅ What Worked:
+
+1. **Coordinate fallback DID activate** (after fix from >= 3 to >= 5)
+2. **Agent successfully used coordinates** at (87.5%, 23.438%)
+3. **Physical clicks succeeded** - page.mouse.click(1120, 169)
+4. **Agent learned** to stick with coordinates after selectors failed
+
+### ❌ What Didn't Work:
+
+**Agent hit max iterations (8) without marking "complete"**
+
+## Detailed Step 6 Flow:
+
+```
+Iteration 1: Selector attempt → Timeout ❌
+Iteration 2: Selector attempt → Timeout ❌
+Iteration 3: Selector attempt → Timeout ❌
+Iteration 4: 🎯 COORDINATE MODE → Click (87.5%, 23.438%) → ✅ Success
+Iteration 5: Repeat coordinate → ✅ Success
+Iteration 6: Repeat coordinate → ✅ Success (?)
+Iteration 7: Repeat coordinate → ✅ Success
+Iteration 8: Repeat coordinate → ✅ Success
+Result: ⚠️ Max iterations → system_limit
+```
+
+## Root Cause Analysis:
+
+### Problem: **No Goal Verification After Coordinate Success**
+
+**With selectors:**
+```typescript
+await page.getByRole('button').click();
+// Can verify: await expect(button).toHaveState('pressed')
+// Can check: New elements appeared, URL changed, etc.
+```
+
+**With coordinates:**
+```typescript
+await page.mouse.click(1120, 169);
+// ❓ Did it work? No element reference!
+// ❓ How to verify? Can't check button state
+// ❓ What changed? Need to inspect DOM/screenshot
+```
+
+### Why Agent Kept Retrying:
+
+**Agent's reasoning (iterations 5-8):**
+- "Coordinate click succeeded (executed without error)"
+- "But I don't know if goal was achieved"
+- "Step says 'Click on New' - did the New form open?"
+- "I should try again to be sure..."
+- → **Loops until max iterations**
+
+## Solutions to Consider:
+
+### Option 1: **Trust Coordinate Success** (Simple)
+After coordinate click succeeds:
+- Wait 500ms for UI response
+- Mark status="complete" automatically
+- Assume click worked (trust the coordinates)
+
+```typescript
+if (coordinateAction && coordResult.allSucceeded) {
+  await page.waitForTimeout(500); // Let UI respond
+  return { status: 'complete', reasoning: 'Coordinate click succeeded' };
+}
+```
+
+**Pros**: Simple, fast
+**Cons**: No verification of actual goal achievement
+
+### Option 2: **Visual Verification** (Better)
+After coordinate click:
+- Wait 500ms
+- Take screenshot
+- Compare before/after
+- If changed → complete, else → retry with different coords
+
+```typescript
+const beforeScreenshot = await page.screenshot();
+await page.mouse.click(x, y);
+await page.waitForTimeout(500);
+const afterScreenshot = await page.screenshot();
+if (screenshotsAreDifferent(before, after)) {
+  return { status: 'complete' };
+}
+```
+
+**Pros**: Validates something changed
+**Cons**: Slower, more LLM calls
+
+### Option 3: **DOM Change Detection** (Balanced)
+After coordinate click:
+- Capture DOM snapshot before
+- Click coordinates
+- Capture DOM snapshot after
+- If new elements/navigation → complete
+
+```typescript
+const beforeUrl = page.url();
+const beforeElements = await getEnhancedPageInfo(page);
+await page.mouse.click(x, y);
+await page.waitForTimeout(500);
+const afterUrl = page.url();
+const afterElements = await getEnhancedPageInfo(page);
+
+if (afterUrl !== beforeUrl || afterElements.count !== beforeElements.count) {
+  return { status: 'complete', reasoning: 'Page state changed after coordinate click' };
+}
+```
+
+**Pros**: Fast, objective verification
+**Cons**: Might miss subtle changes (modal opens without URL/element count change)
+
+### Option 4: **Prompt Guidance** (Immediate)
+Update prompt to tell agent:
+"After coordinate click succeeds, mark status='complete' unless you can clearly verify it failed"
+
+**Pros**: No code changes
+**Cons**: Relies on LLM judgment
+
+## Recommendation:
+
+**Hybrid approach:**
+1. **Immediate** (Prompt): Tell agent to trust coordinate success
+2. **Phase 2** (Code): Add DOM change detection for validation
+
+## Current Status:
+
+- ✅ Coordinate fallback works technically
+- ✅ Physical clicks succeed
+- ❌ Agent doesn't know when to stop
+- 🔧 Need completion detection logic
+
+## Test Results Summary:
+
+**Steps 1-5**: ✅ All completed successfully
+**Step 6**: ⚠️ Coordinates worked but hit max iterations (no completion detection)
+**Overall**: Coordinate mode is functional but needs completion logic
+

package/plandocs/IMPLEMENTATION_STATUS.md

@@ -0,0 +1,108 @@
+# Runner-Core Visual Agent Implementation Status
+
+## Phase 1: ✅ COMPLETE (v0.0.33)
+
+### Implemented Features:
+
+1. **Note to Future Self** - Tactical iteration memory
+2. **Percentage-Based Coordinates** - Last-resort fallback with 3-decimal precision
+3. **Two-Tier Auto-Escalation** - Code-controlled mode switching
+
+### Current Behavior (Phase 1):
+
+```
+Iteration 1-3: Normal Playwright selectors + note-to-self (3 attempts)
+    ↓ (after 3 failures)
+Iteration 4-5: Percentage coordinates (2 attempts max)
+    ↓ (if both coordinate attempts fail)
+Give up - mark as stuck
+
+Total: Maximum 5 iterations per step
+```
+
+---
+
+## Phase 2: 📋 PLANNED (Not Started)
+
+### Will Add:
+
+1. **ElementDetector** - Detect interactive elements with z-index awareness
+2. **VisualMarkerInjector** - Number elements [1], [2], [3] on screenshot
+3. **SelectorResolver** - Translate index → native Playwright selector
+4. **IndexCommandTranslator** - Convert CLICK[3] → native Playwright command
+
+### Future Behavior (Phase 2):
+
+```
+Iteration 1: Playwright selector (1 attempt) → 70% success
+    ↓ (on first failure)
+Iteration 2-3: Index commands CLICK[3] (2 attempts) → 25% success
+    ↓ (after 3 total failures)
+Iteration 4-5: Percentage coordinates (2 attempts max) → 5% success
+    ↓ (if all fail)
+Give up - mark as stuck
+
+Total: Maximum 5 iterations per step (down from 8)
+Average: ~1.5 iterations per step (fast!)
+```
+
+### Key Design Principle for Phase 2:
+
+**During Execution:**
+- Agent clicks using `data-testchimp-el="[3]"` (reliable, we inject it)
+
+**In Generated Script:**
+- Translator outputs NATIVE selector: `getByRole('button', {name: 'Menu'})`
+- Script works standalone without data-testchimp-el
+
+**Why Two-Stage:**
+1. Agent needs reliability during exploration → use data attribute
+2. Generated script must be portable → use native selectors
+3. Best of both worlds: reliable execution + maintainable output
+
+---
+
+## Optimizations vs Original Plan
+
+### Original Plan:
+- Tier 1: iterations 1-2
+- Tier 2: iterations 3-4
+- Tier 3: iterations 5+
+- Average: ~4 iterations per step
+
+### Optimized Plan (Current):
+- Tier 1: iteration 1 ONLY (fast path)
+- Tier 2: iterations 2-3 (reliable fallback)
+- Tier 3: iterations 4+ (absolute last resort)
+- **Target: ~1.5 average iterations per step**
+
+**Rationale:** Don't waste time! Simple tasks finish in 1 iteration, complex tasks escalate quickly to more reliable methods.
+
+---
+
+## Testing Checklist
+
+### Phase 1 (Ready Now):
+- [ ] Run PeopleHR scenario - verify note-to-self helps
+- [ ] Test coordinate fallback on deliberately difficult case
+- [ ] Measure iteration reduction (expect 20-30%)
+- [ ] Verify timeout fixes for waitForLoadState
+
+### Phase 2 (When Implemented):
+- [ ] Test ElementDetector on modals/overlays
+- [ ] Verify z-index occlusion detection
+- [ ] Validate native selector generation (no data-testchimp-el in output)
+- [ ] Run generated scripts standalone - must work!
+- [ ] Measure tier distribution: 70/25/5
+
+---
+
+## Current Version
+
+**Runner-Core:** v0.0.33  
+**Status:** Built and ready to test  
+**Phase 1:** ✅ Complete  
+**Phase 2:** 📋 Planned but not started  
+
+**Next Step:** Test Phase 1 with PeopleHR scenario to validate improvements before implementing Phase 2.
+

package/plandocs/PHASE_1_COMPLETE.md

@@ -0,0 +1,165 @@
+# Phase 1 Implementation - COMPLETE ✅
+
+## Version: runner-core v0.0.33
+
+## What's Been Implemented
+
+### 1. Free-Form "Note to Future Self" 
+**Purpose:** Tactical memory - agent leaves notes that persist across iterations AND steps.
+
+**Type:**
+```typescript
+interface NoteToFutureSelf {
+  fromIteration: number;
+  content: string;  // FREE-FORM - agent writes whatever it wants
+}
+```
+
+**How it works:**
+- Agent includes `"noteToFutureSelf": "..."` in response
+- System stores it in `memory.latestNote` (persists across steps!)
+- Passed to next iteration AND next step
+- Displayed prominently at top of prompt
+- Agent reads it FIRST before making decision
+
+**Scope:** Entire scenario journey (not just current step)
+
+**Example notes:**
+
+*Iteration-specific:*
+- "Tried #sidebar-toggle, failed with 'not clickable'. Will try child SVG element next."
+
+*Step-spanning:*
+- "This app has slow-loading modals. Always wait 2s after page load before clicking."
+- "Cookie consent appears on every page. Check for and dismiss it first."
+- "Sidebar only visible on desktop viewport (>1024px width)."
+
+### 2. Percentage-Based Coordinate Fallback
+**Purpose:** Last-resort mechanism when selector generation repeatedly fails.
+
+**Type:**
+```typescript
+interface CoordinateAction {
+  type: 'coordinate';
+  action: 'click' | 'doubleClick' | 'rightClick' | 'hover' | 'drag' | 'fill' | 'scroll';
+  xPercent: number;  // 0-100, 3 decimal precision
+  yPercent: number;
+  toXPercent?: number;  // For drag
+  toYPercent?: number;
+  value?: string;  // For fill
+  scrollAmount?: number;  // For scroll
+}
+```
+
+**How it works:**
+- LLM outputs percentages: `{xPercent: 15.755, yPercent: 8.500}`
+- CoordinateConverter converts to pixels: `15.755% → 252px`
+- Generates Playwright command: `await page.mouse.click(252, 68);`
+
+**Supported actions:**
+- click, doubleClick, rightClick, hover
+- fill (clicks then types value)
+- drag (from x%,y% to toX%,toY%)
+- scroll (at position, by amount)
+
+### 3. Two-Tier Auto-Escalation
+**Trigger:** Code-controlled (not LLM-decided)
+
+```
+Tier 1 (iterations 1-3): Playwright Selector Mode
+├─ Normal buildSystemPrompt()
+├─ Agent generates: await page.getByRole(...).click()
+├─ Leaves noteToFutureSelf for continuity
+└─ 3 attempts, then escalate
+
+Tier 2 (iterations 4-5): Coordinate Mode
+├─ Auto-activates when consecutiveFailures >= 3
+├─ Uses buildCoordinateSystemPrompt()
+├─ Agent outputs: {xPercent: 15.755, yPercent: 8.500}
+├─ CoordinateConverter → mouse.click(x, y)
+└─ 2 attempts max, then give up
+
+Total: Maximum 5 iterations per step
+```
+
+### 4. Precision & Accuracy
+- **3 decimal precision** for coordinates (~1px accuracy on most screens)
+- **Resolution-independent** - works on any viewport size
+- **Percentage reference:**
+  - Top-left: (0, 0)
+  - Top-right: (100, 0)
+  - Center: (50, 50)
+  - Bottom-right: (100, 100)
+
+## Files Modified
+
+1. **orchestrator/types.ts**
+   - Added `NoteToFutureSelf` interface
+   - Added `CoordinateAction` interface
+   - Updated `AgentDecision` with new fields
+   - Updated `AgentContext` with noteFromPreviousIteration
+
+2. **orchestrator/orchestrator-agent.ts**
+   - Added note tracking in executeStep()
+   - Added coordinate action execution
+   - Added buildCoordinateSystemPrompt()
+   - Updated buildUserPrompt() to display notes
+   - Added mode switching in callAgent()
+   - Updated response format documentation
+
+3. **utils/coordinate-converter.ts** (NEW)
+   - percentToPixels() - Convert % to pixels
+   - getViewportSize() - Get current viewport dimensions
+   - generateCommands() - Create Playwright commands from percentages
+   - executeAction() - Direct execution helper
+
+4. **scenario-worker-class.ts** (Earlier fix)
+   - Smart timeout handling for waitForLoadState
+
+5. **execution-service.ts** (Earlier fix)
+   - Smart timeout handling for navigation commands
+
+## How to Use
+
+**No code changes needed!** The features activate automatically:
+
+1. **Note to self:** Agent can optionally include `noteToFutureSelf` in any iteration
+2. **Coordinates:** Auto-activate at iteration 4 if selectors keep failing
+
+## Testing Phase 1
+
+To validate the implementation:
+
+1. **Run PeopleHR scenario** (previously failed on hamburger menu)
+   - Should now succeed with note guidance
+   - May use coordinates if SVG selector still fails
+
+2. **Check logs for:**
+   - `📝 Note to self: ...` (agent leaving tactical notes)
+   - `🎯 COORDINATE MODE ACTIVATED` (tier 2 triggered)
+   - `🎯 Coordinate Action: click at (X%, Y%)` (using fallback)
+
+3. **Expected improvements:**
+   - 20-30% fewer iterations per step (thanks to notes)
+   - < 5% scenarios need coordinate fallback
+   - Coordinates work when everything else fails
+
+## Phase 2 Preview (Not Yet Implemented)
+
+When Phase 2 is added, it will become a **three-tier** system:
+- Tier 1 (iterations 1-2): Playwright selectors
+- Tier 2 (iterations 3-4): Numbered elements (CLICK[3])
+- Tier 3 (iterations 5+): Percentage coordinates
+
+Phase 2 adds visual markers [1], [2], [3] on elements with structured commands.
+
+---
+
+## Status: ✅ READY FOR TESTING
+
+Runner-core v0.0.33 is built and ready. Test it with:
+- VS Code extension "Run Test" on peoplehr-corrected.smart.spec.ts
+- Or generate new script from peoplehr.txt scenario
+
+**Next:** Validate Phase 1 works before starting Phase 2.
+