testchimp-runner-core 0.0.32 → 0.0.34

package/plandocs/SESSION_SUMMARY_v0.0.33.md

@@ -0,0 +1,151 @@
+# Runner-Core v0.0.33 - Session Summary
+
+## Date: October 15, 2025
+
+## Major Accomplishments:
+
+### 1. ✅ **Coordinate Fallback System** (Phase 1 Complete)
+- Percentage-based coordinates (0-100%, 3 decimal precision)
+- Activates after 3 selector failures
+- 2 coordinate attempts before giving up
+- Resolution-independent positioning
+
+### 2. ✅ **Note to Future Self** (Tactical Memory)
+- Free-form notes persist across iterations AND steps
+- Enables strategic planning across agent decisions
+- Helps maintain context: "Tried X, will try Y next"
+
+### 3. ✅ **Visual Verification Tool** (NEW)
+- `verify_action_result` - Before/after screenshot comparison
+- Agent-callable (decides when to use)
+- JPEG 60% quality (85-90% smaller than PNG)
+- Multi-image LLM interface support
+
+### 4. ✅ **Critical Bug Fixes**
+- **Coordinate mode never activated**: Changed forced stuck from >= 3 to >= 5 failures
+- **Missing required fields**: Made parser flexible (accepts reasoning OR statusReasoning)
+- **Navigation timeouts**: Added 30s timeout guidance for page.goto()
+- **Strict mode violations**: Added scoping guidance (locator('#parent').getByText())
+
+### 5. ✅ **Prompt Optimizations**
+- **59% reduction**: 17,573 chars → 7,287 chars in system prompt
+- **Cache-optimized**: Static content first, dynamic last
+- **Cost savings**: ~40% overall with model tiering
+- **Focused on cognition**: Removed bloat, kept decision-making guidance
+
+### 6. ✅ **Model Optimization**
+- **gpt-5-mini**: Complex tasks (4 operations)
+  - Command generation
+  - Goal completion checks
+  - Repair suggestions
+  - Agent orchestration
+- **gpt-4o-mini**: Simple tasks (7 operations)
+  - Scenario breakdown
+  - Screenshot need assessment
+  - Repair confidence
+  - Test name generation
+  - Hashtag generation
+  - Script parsing
+  - Final script merging
+- **Est. 25-30% cost reduction**
+
+### 7. ✅ **Code Cleanup**
+- Removed V1 SmartTestRunnerCore (V2 is stable)
+- Removed backup files (.bak, .tmp)
+- Consolidated types into V2
+- Removed PeopleHR-specific examples from prompts
+
+### 8. ✅ **Enhanced Logging**
+- Prompt length metrics (chars + estimated tokens)
+- Full LLM response on parsing errors
+- Field presence diagnostics
+- Retry logging for 500 errors
+
+### 9. ✅ **Retry Logic**
+- Automatic retry for OpenAI 500 errors
+- Exponential backoff (1s, 2s, 4s)
+- Up to 3 attempts before failing
+
+### 10. ✅ **Headed Mode for Local Testing**
+- All browser instances use headed: false → headed: false for local dev
+- Visual debugging enabled
+
+## Files Modified:
+
+### Runner-Core:
+1. `src/orchestrator/orchestrator-agent.ts` - Main agent logic
+2. `src/orchestrator/types.ts` - NoteToFutureSelf, CoordinateAction
+3. `src/utils/coordinate-converter.ts` - NEW - Coordinate to Playwright conversion
+4. `src/orchestrator/tools/verify-action-result.ts` - NEW - Visual verification tool
+5. `src/llm-provider.ts` - Added LabeledImage, multi-image support
+6. `src/llm-facade.ts` - Model optimization
+7. `src/model-constants.ts` - Added DEFAULT_SIMPLER_MODEL
+8. `src/scenario-worker-class.ts` - Tool registration
+9. `src/orchestrator/index.ts` - Exports
+10. `src/orchestrator/tools/index.ts` - Tool exports
+
+### Scriptservice:
+1. `providers/scriptservice-llm-provider.ts` - Multi-image handling, retry logic
+2. `smart-test-runner-core-v2.ts` - Type definitions, V1 removal
+3. `smart-test-execution-handler.ts` - V1 removal
+4. `workers/test-based-explorer.ts` - V1 removal
+5. `script-generation-handlers.ts` - Headed mode
+6. `script-generation/script-generation-service.ts` - Headed mode
+7. `smart-test-execution-handler.ts` - Headed mode
+
+### Documentation:
+1. `WHATS_NEW_v0.0.33.md`
+2. `PHASE_1_COMPLETE.md`
+3. `PHASE_1_SUMMARY.md`
+4. `IMPLEMENTATION_STATUS.md`
+5. `VISUAL_AGENT_EVOLUTION_PLAN.md`
+6. `PROMPT_SANITY_CHECK.md`
+7. `PROMPT_OPTIMIZATION_ANALYSIS.md`
+8. `COORDINATE_MODE_DIAGNOSIS.md`
+9. `BEFORE_AFTER_VERIFICATION.md`
+10. `TROUBLESHOOTING_SESSION.md`
+
+## Live Test Status:
+
+**Job**: `71b88c60-52f5-4343-aef8-c44ebb07f3e9`
+**Status**: Running (check browser + logs)
+**Watch For**:
+- Step 5 (Employee Information) - Previously problematic
+- Coordinate mode activation
+- verify_action_result tool usage
+- Overall completion
+
+## Key Metrics:
+
+**Cost Optimization:**
+- Prompt size: 59% reduction
+- Model tiering: 7/11 tasks on cheaper model
+- JPEG compression: 85-90% smaller screenshots
+- **Total savings: ~40% cost reduction**
+
+## Next Steps After Test Completes:
+
+1. Check if Step 5 completes successfully
+2. Verify coordinate mode activated if needed
+3. Check if verify_action_result tool was used
+4. Analyze any remaining failures
+5. Iterate on prompts/logic based on results
+
+## Known Issues to Monitor:
+
+1. **Step 5 False Positive**: Clicking menu item vs navigating to page
+2. **Coordinate Loop**: Agent not knowing when coordinate clicks succeed
+3. **Vision verification usage**: Will agent call it proactively?
+
+## Success Criteria:
+
+✅ All 7 steps complete
+✅ Coordinate fallback used when selectors fail
+✅ Visual verification validates goal achievement
+✅ No infinite loops or stuck states
+✅ Generated script is accurate
+
+---
+
+**Check your browser window and /tmp/scriptservice-test.log for live execution!**
+

package/plandocs/TROUBLESHOOTING_SESSION.md

@@ -0,0 +1,72 @@
+# Troubleshooting Session: All Modules Icon Click Failure
+
+## Objective:
+Understand why the orchestrator agent gets stuck on "Click on the all Modules menu item (top menu icon)" while manual Playwright MCP navigation succeeded.
+
+## What I Need to See:
+
+### 1. Full Agent Logs for the Failing Step
+Please provide the complete logs showing:
+- What iteration attempts were made (iteration 1, 2, 3...)
+- What selectors the agent tried each time
+- What errors it encountered
+- What the DOM snapshot showed
+- Whether it took screenshots
+- What notes it left to future self
+
+### 2. The DOM Context It Saw
+- Interactive elements list
+- ARIA tree snapshot
+- Whether the hamburger icon was visible in the list
+
+## What Worked (My Manual MCP Session):
+
+From earlier successful navigation:
+```
+✅ Step 1: Clicked hamburger menu
+   Selector: #sidebar-toggle > span > svg
+   
+✅ Step 2: Clicked "Core HR"  
+   Selector: getByText('Core HR')
+   
+✅ Step 3: Clicked "Employee Information"
+   Selector: getByText('Employee Information')
+```
+
+## Hypothesis of Why Agent Fails:
+
+### Possible Issue 1: Wrong Selector Strategy
+- Agent might be trying: `getByText('All Modules')` (strict mode violation)
+- Or: `#MenuToggle` (wrong ID)
+- Or: `#sidebar-toggle-menu` (doesn't exist)
+- Instead of: `#sidebar-toggle > span > svg` (actual selector)
+
+### Possible Issue 2: Missing Icon Detection
+- Hamburger icons are often SVG elements without accessible text
+- Agent might not recognize this pattern
+- Prompt doesn't explicitly guide on icon/SVG selector strategy
+
+### Possible Issue 3: DOM List Incomplete
+- Interactive elements might not include the SVG icon
+- If icon isn't in the list, agent won't know it exists
+- Need to check if `getEnhancedPageInfo` captures SVG icons
+
+### Possible Issue 4: Ambiguous Text
+- "All Modules" might appear in multiple places (menu button + modal title)
+- Agent tries `getByText('All Modules')` → strict mode violation
+- Should scope to parent: `locator('#sidebar-toggle').getByText('All Modules')`
+
+## Next Steps:
+
+1. **Get full logs** from your failing run
+2. **Compare** what agent saw vs what I saw  
+3. **Identify** the gap (prompt, DOM extraction, or selector logic)
+4. **Plan fixes**:
+   - Prompt improvements (icon/SVG guidance)
+   - DOM extraction improvements (ensure icons are captured)
+   - Selector strategy improvements (parent scoping for icons)
+   - Example-based learning (hamburger menu pattern)
+
+## Waiting For:
+Please paste the full logs from the failing step showing all iteration attempts and what the agent tried.
+

package/plandocs/VISUAL_AGENT_EVOLUTION_PLAN.md

@@ -0,0 +1,396 @@
+# Runner-Core Visual Agent Evolution - Complete Implementation Plan
+
+## Overview
+
+Two-phase pragmatic evolution without major architecture overhaul:
+- **Phase 1 (Week 1-2):** Percentage coordinates + Free-form notes
+- **Phase 2 (Week 3-5):** Numbered element system with three-tier fallback
+
+---
+
+## PHASE 1: Tactical Improvements
+
+### 1A: Free-Form "Note to Future Self"
+
+**Why:** Agent needs tactical memory between iterations of the SAME step.
+
+**Type:**
+```typescript
+interface NoteToFutureSelf {
+  fromIteration: number;
+  content: string;  // FREE-FORM - agent writes whatever it wants
+}
+```
+
+**Examples:**
+- "Tried #sidebar-toggle, failed. Will try SVG child next."
+- "Plan: Hover over menu first to reveal dropdown, then click Settings."
+- "Cookie banner blocking. Next: dismiss it, then retry main action."
+
+**vs Current Learnings:**
+- Learnings = App-wide patterns ("App uses getByRole")
+- Note to self = Iteration-specific tactics ("Just tried X, will try Y next")
+
+**Keep BOTH!**
+
+### 1B: Percentage-Based Coordinates
+
+**LLM outputs percentages:**
+```json
+{
+  "coordinateAction": {
+    "action": "click|fill|drag|hover|scroll",
+    "xPercent": 15.75,  // 2 decimal precision for accuracy
+    "yPercent": 8.50,
+    
+    // For drag:
+    "toXPercent": 45.25,
+    "toYPercent": 8.50,
+    
+    // For fill:
+    "value": "alice@example.com"
+  }
+}
+```
+
+**Code converts to pixels:**
+```typescript
+const viewport = await page.evaluate(() => ({width: window.innerWidth, height: window.innerHeight}));
+const x = Math.round((xPercent / 100) * viewport.width);
+const y = Math.round((yPercent / 100) * viewport.height);
+```
+
+### 1C: Code-Controlled Fallback
+
+**Tier 1 (iteration 1-3):** Normal Playwright selectors
+**Tier 2 (iteration 4+):** Percentage coordinates (auto-triggered after 3 failures)
+
+```typescript
+// In callAgent():
+if (consecutiveFailures >= 3) {  // Phase 1: Only 2 tiers
+  // Auto-use coordinate-specific system prompt
+  // LLM outputs percentages
+  // Phase 2 will add tier 2 (indexed elements) between selectors and coordinates
+}
+```
+
+**Files:**
+- `utils/coordinate-converter.ts` (NEW)
+- `orchestrator/orchestrator-agent.ts` (UPDATE)
+- `orchestrator/types.ts` (UPDATE - add CoordinateAction, NoteToFutureSelf)
+
+---
+
+## PHASE 2: Numbered Element System
+
+### Architecture
+
+```
+Three-Tier Fallback:
+
+Tier 1 (iteration 1 ONLY): Playwright Selector Mode - ONE SHOT
+  ├─ Agent generates: await page.getByRole('button', {name: 'Login'}).click()
+  ├─ Direct execution
+  └─ 70% of tasks finish here (simple/medium complexity)
+
+Tier 2 (iterations 2-3): Index Command Mode - TWO ATTEMPTS
+  ├─ Inject numbered markers [1], [2], [3] → screenshot
+  ├─ Agent outputs: CLICK[3], FILL[5, "alice@example.com"]
+  ├─ Execution: Use data-testchimp-el="[3]" (reliable targeting)
+  ├─ Script output: Translate to NATIVE selector (getByRole, #id, etc.)
+  └─ 25% of tasks finish here (complex UIs, icons, shadow DOM)
+
+Tier 3 (iterations 4+): Percentage Coordinate Mode - LAST RESORT
+  ├─ Agent outputs: {xPercent: 15.755, yPercent: 8.500}
+  ├─ CoordinateConverter: % → pixels
+  ├─ Execute: mouse.click(x, y)
+  └─ <5% of tasks need this (extreme edge cases)
+```
+
+### 2.1: Reusable Utility: ElementDetector
+
+**File:** `runner-core/src/utils/element-detector.ts`
+
+**Purpose:** Detect ALL interactive elements with z-index and occlusion awareness.
+
+**Key Features:**
+- Comprehensive element queries (buttons, links, inputs, SVGs, clickable divs)
+- Z-index calculation via `getComputedStyle()`
+- Occlusion detection via `elementFromPoint()` at center
+- Context tags: `[header|nav|sidebar|main|modal]`
+- Spatial tags: `[top|bottom|left|right|center]`
+- Center position as percentage (2 decimal precision)
+
+**Output:**
+```typescript
+interface DetectedElement {
+  index: number;                    // [1], [2], [3]
+  tag: string;
+  text: string;
+  role: string;
+  ariaLabel: string;
+  bbox: {x, y, width, height};
+  centerPercent: {x: 15.75, y: 8.50};  // As percentage
+  context: string[];                // ['header', 'nav', 'top']
+  zIndex: number;
+  isVisible: boolean;               // Not occluded by higher z-index
+  selectors: {
+    dataAttribute: string;          // [data-testchimp-el="[3]"]
+    semantic: string[];             // [getByRole(...), getByLabel(...)]
+    cssId: string | null;
+    cssClass: string | null;
+  };
+}
+```
+
+**Implementation Highlights:**
+- Queries: `button`, `a[href]`, `input`, `svg`, `[onclick]`, `[role=...]`, `[data-testid]`, clickable divs/spans
+- Z-index check: `elementFromPoint(centerX, centerY)` must return element or child
+- Filter: Only return `isVisible: true` elements
+
+### 2.2: Reusable Utility: SelectorResolver
+
+**File:** `runner-core/src/utils/selector-resolver.ts`
+
+**Purpose:** Given element index, return most reliable Playwright selector FOR GENERATED SCRIPTS.
+
+**CRITICAL DISTINCTION:**
+
+A. **During Execution (Internal):**
+   - Agent can click using `data-testchimp-el="[N]"` (we inject it temporarily)
+   - This ensures agent clicks the exact right element
+
+B. **For Generated Script (Output):**
+   - Must use NATIVE selectors that work without our attributes
+   - Script will run on real application without data-testchimp-el
+
+**Selector Resolution for Script Output (in order):**
+1. Semantic selectors - `getByRole()`, `getByLabel()` (BEST - maintainable)
+2. CSS ID - `#element-id` (GOOD - stable)
+3. CSS class - `.button-primary` (scoped to context if ambiguous)
+4. Contextual selector - `header .menu-toggle svg` (LAST RESORT)
+
+**For each selector, validate:**
+- Element exists on page
+- Not covered by higher z-index (using elementFromPoint)
+- Position matches expected (±5% tolerance)
+
+**Method:**
+```typescript
+static async resolveIndexToSelector(
+  index: number,
+  elements: DetectedElement[],
+  page: Page
+): Promise<{selector: string, strategy: string}>
+```
+
+**Validation:**
+```typescript
+private static async validateInteractable(
+  page: Page,
+  selector: string,
+  expectedCenterPercent: {x, y}
+): Promise<boolean> {
+  // 1. Element exists
+  // 2. Non-zero dimensions
+  // 3. elementFromPoint(center) === element (z-index check)
+  // 4. Position within ±5% tolerance
+}
+```
+
+### 2.3: Reusable Utility: VisualMarkerInjector
+
+**File:** `runner-core/src/utils/visual-marker-injector.ts`
+
+**Purpose:** Inject visual numbered labels on page.
+
+**Methods:**
+```typescript
+static async injectNumberedMarkers(page): Promise<DetectedElement[]>
+static async removeMarkers(page): Promise<void>
+static async captureMarkedScreenshot(page, elements): Promise<string>
+```
+
+**Visual Markers:**
+- Red gradient background `[1]`, `[2]`, `[3]`
+- Positioned at top-left of element
+- z-index: 999999 (always visible)
+- Inject `data-testchimp-el` attribute on actual element (TEMPORARY - for agent execution only)
+  * Used internally to ensure agent clicks correct element
+  * Removed after step completion
+  * NEVER appears in generated script output
+
+### 2.4: Reusable Utility: IndexCommandTranslator
+
+**File:** `runner-core/src/utils/index-command-translator.ts`
+
+**Purpose:** Translate index commands to Playwright commands with NATIVE selectors (for script generation).
+
+**Input:**
+```typescript
+{ action: "CLICK", index: 3 }
+{ action: "FILL", index: 5, value: "alice@example.com" }
+```
+
+**Output (MUST use native selectors):**
+```typescript
+"await page.getByRole('button', {name: 'Menu'}).click();"
+"await page.locator('#username').fill('alice@example.com');"
+// OR
+"await page.locator('#sidebar-toggle svg').click();"
+```
+
+**NOT this (won't work in generated script):**
+```typescript
+❌ "await page.locator('[data-testchimp-el=\"[3]\"]').click();"
+❌ "await page.locator('[data-testchimp-el=\"[5]\"]').fill('alice@example.com');"
+```
+
+**Process:**
+1. **During execution:** Click element using `data-testchimp-el="[index]"` (reliable)
+2. **For script output:** Use SelectorResolver to get NATIVE selector (semantic/id/class)
+3. Generate Playwright command with native selector
+4. Return command string that works on real application
+
+**Critical Distinction:**
+- `data-testchimp-el` = Internal execution helper (temporary)
+- Script output = Native selectors (permanent, works standalone)
+
+### 2.5: Integration - Three-Tier System (Optimized Escalation)
+
+**File:** `orchestrator/orchestrator-agent.ts`
+
+**Optimized Strategy:** Escalate quickly to avoid wasting time on difficult tasks
+
+**Mode Determination:**
+```typescript
+let tier: 1 | 2 | 3;
+
+// Tier 1: Try normal selectors ONCE (iteration 1)
+// Tier 2: Use indexed elements TWICE (iterations 2-3) 
+// Tier 3: Use percentage coords (iterations 4+)
+
+if (iteration >= 4) {
+  tier = 3;  // Coordinate mode
+} else if (iteration >= 2) {
+  tier = 2;  // Index command mode
+} else {
+  tier = 1;  // Normal Playwright selector mode
+}
+```
+
+**Rationale:**
+- Simple tasks: Succeed in Tier 1 (iteration 1) - fast!
+- Medium tasks: Tier 2 gives 2 attempts with reliable index system
+- Hard tasks: Tier 3 coordinates as absolute fallback
+- No wasted iterations on difficult element detection
+
+**Tier Preparation:**
+```typescript
+if (tier === 2) await this.prepareIndexMode(context, page);
+if (tier === 3) await this.prepareCoordinateMode(context, page);
+```
+
+**System Prompt Selection:**
+```typescript
+const systemPrompt = 
+  tier === 3 ? this.buildCoordinateSystemPrompt() :
+  tier === 2 ? this.buildIndexSystemPrompt() :
+  this.buildSystemPrompt();
+```
+
+**Execution Flow:**
+```typescript
+// Tier 1 (iteration 1):
+if (decision.commands) {
+  // Execute normal Playwright command
+  // If fails → iteration++, move to Tier 2
+}
+
+// Tier 2 (iterations 2-3):
+if (decision.indexCommand) {
+  // Step A: Click using data-testchimp-el="[N]" (execution)
+  await page.locator('[data-testchimp-el="[3]"]').click();
+  
+  // Step B: Resolve to native selector (script generation)
+  const nativeSelector = await SelectorResolver.resolve(3, elements);
+  // → Returns: "getByRole('button', {name: 'Menu'})"
+  
+  // Step C: Add to generated script
+  commandsExecuted.push(`await page.${nativeSelector}.click();`);
+  
+  // If fails after 2 attempts → iteration++, move to Tier 3
+}
+
+// Tier 3 (iterations 4+):
+if (decision.coordinateAction) {
+  // Convert % to pixels and execute
+  // Add coordinate commands to script (acceptable for edge cases)
+}
+```
+
+---
+
+## Utilities Summary
+
+All utilities are **stateless and reusable**:
+
+| Utility | Purpose | Reusable For |
+|---------|---------|--------------|
+| ElementDetector | Find interactive elements | Accessibility audits, page analysis |
+| SelectorResolver | Index → selector with validation | Any numbered system |
+| VisualMarkerInjector | Add visual labels | Manual testing, debugging |
+| IndexCommandTranslator | Index command → Playwright | Any index-based automation |
+| CoordinateConverter | Percentage → pixels | Any coordinate system |
+
+---
+
+## Implementation Timeline
+
+### Week 1: Phase 1 Core
+- [ ] NoteToFutureSelf type and tracking
+- [ ] CoordinateAction with percentages
+- [ ] CoordinateConverter utility
+- [ ] Coordinate mode switching (tier 3)
+
+### Week 2: Phase 1 Testing
+- [ ] Test note-to-self on 10 scenarios
+- [ ] Test percentage coordinates at multiple viewport sizes
+- [ ] Verify all coordinate actions (click, fill, drag, scroll, hover)
+
+### Week 3: Phase 2 Utilities
+- [ ] ElementDetector with z-index awareness
+- [ ] SelectorResolver with occlusion validation
+- [ ] Test utilities standalone on complex pages
+
+### Week 4: Phase 2 Integration
+- [ ] VisualMarkerInjector
+- [ ] IndexCommandTranslator (TWO-STAGE: execution via data-attr, script via native selector)
+- [ ] Index mode (tier 2) integration with iteration-based switching
+- [ ] Optimized escalation: iteration 1 → tier 1, iteration 2-3 → tier 2, iteration 4+ → tier 3
+- [ ] Test PeopleHR with tier 2 (should succeed in iteration 2-3)
+
+### Week 5: Phase 2 Testing
+- [ ] Three-tier end-to-end testing
+- [ ] Measure tier distribution (target: 70/25/5)
+- [ ] A/B test vs current implementation
+- [ ] Performance optimization
+
+## Success Metrics
+
+**Phase 1:**
+- 20-30% reduction in average iterations per step
+- Note-to-self prevents 40%+ of repeated selector failures
+- Coordinates used in < 5% of scenarios
+
+**Phase 2:**
+- 70% scenarios complete in Tier 1 (iteration 1) - simple cases
+- 25% scenarios use Tier 2 (iterations 2-3) - complex UIs with icons/shadows
+- < 5% scenarios escalate to Tier 3 (iterations 4+) - impossible selector cases
+- **PeopleHR hamburger menu:** Succeeds in Tier 2 iteration 2 with CLICK[N]
+- **Average iterations per step:** Should decrease from ~4 to ~1.5
+
+## Total Effort: 4-5 weeks
+
+**Ready to implement?**
+