testchimp-runner-core 0.0.33 → 0.0.35

package/plandocs/PHASE_1_SUMMARY.md

@@ -0,0 +1,184 @@
+# Phase 1 Complete - Summary & Testing Guide
+
+## Version: runner-core v0.0.33 ✅
+
+---
+
+## Implementation Complete
+
+### What's New:
+
+1. **📝 Note to Future Self**
+   - Free-form tactical memory between iterations
+   - Agent writes: "Tried X, failed. Will try Y next."
+   - Prevents repeated mistakes
+
+2. **🎯 Percentage-Based Coordinates**  
+   - Last-resort fallback (3-decimal precision)
+   - Resolution-independent (works any viewport size)
+   - Supports: click, fill, drag, hover, scroll
+
+3. **⚡ Optimized Iteration Limits**
+   - Max 5 iterations per step (down from 8)
+   - 2 coordinate attempts max (coordinates work or they don't)
+   - Faster feedback on stuck scenarios
+
+---
+
+## Current Behavior (Phase 1)
+
+```
+┌─────────────────────────────────────────────────────┐
+│  Iteration 1: Playwright selector                   │
+│    Try: await page.getByRole('button'...).click()   │
+│    Note: "If this fails, try #id selector"          │
+│                                                      │
+│  Iteration 2: Playwright selector                   │
+│    Read note from iteration 1                       │
+│    Try: await page.locator('#sidebar-toggle').click()│
+│    Note: "If this fails, try SVG child"             │
+│                                                      │
+│  Iteration 3: Playwright selector                   │
+│    Read note from iteration 2                       │
+│    Try: await page.locator('#sidebar-toggle svg')   │
+│    → Fails again                                     │
+│                                                      │
+│  🎯 COORDINATE MODE ACTIVATED 🎯                     │
+│                                                      │
+│  Iteration 4: Coordinate action                     │
+│    Agent outputs: {xPercent: 5.500, yPercent: 8.250}│
+│    Execute: page.mouse.click(88, 66)                │
+│    → Success!                                        │
+│                                                      │
+│  OR if fails...                                      │
+│                                                      │
+│  Iteration 5: Coordinate action (2nd attempt)       │
+│    Try slightly adjusted coordinates                 │
+│    → If fails: GIVE UP (stuck)                      │
+│                                                      │
+│  Total: Max 5 iterations                            │
+└─────────────────────────────────────────────────────┘
+```
+
+---
+
+## Testing Phase 1
+
+### Test 1: PeopleHR Scenario (Previously Failed)
+
+**Expected outcome:**
+- Iteration 1-2: Try text/ID selectors → fail
+- Iteration 3: Note says "try SVG child" → succeeds!
+- OR Iteration 4: Coordinates → succeeds!
+
+**Run:**
+```bash
+# Via VS extension: "Generate Script" on peoplehr.txt
+# Or "Run Test" on peoplehr-corrected.smart.spec.ts
+```
+
+**Look for in logs:**
+```
+📝 Note to self: ...
+🎯 COORDINATE MODE ACTIVATED
+🎯 Coordinate Action (attempt 1/2): click at (5.500%, 8.250%)
+```
+
+### Test 2: Simple Scenario (Should Still Be Fast)
+
+Create test: `simple-login.txt`
+```
+- go to https://example.com/login
+- fill username with "alice"
+- fill password with "password123"
+- click login button
+```
+
+**Expected:**
+- Each step: 1 iteration (Tier 1 success)
+- No coordinates needed
+- Fast execution
+
+### Test 3: Coordinate Fallback
+
+**Deliberately difficult scenario:**
+```
+- go to https://some-app-with-shadow-dom.com
+- click on custom web component icon
+```
+
+**Expected:**
+- Iterations 1-3: Selectors fail
+- Iteration 4: Coordinates succeed
+- Generated script contains: `await page.mouse.click(x, y);`
+
+---
+
+## Expected Improvements
+
+### Metrics to Track:
+
+1. **Iteration Efficiency**
+   - Before: ~4 average iterations per step
+   - After: ~2.5 average iterations per step (30-40% reduction)
+
+2. **Success Rate**
+   - Before: Stuck on complex UIs (hamburgers, icons, shadow DOM)
+   - After: Coordinates provide escape hatch
+
+3. **Coordinate Usage**
+   - Target: < 10% of scenarios use coordinates
+   - Most scenarios still succeed with selectors
+
+---
+
+## Files Changed
+
+**New:**
+- `src/utils/coordinate-converter.ts` - Percentage conversion utility
+- `VISUAL_AGENT_EVOLUTION_PLAN.md` - Complete plan
+- `PHASE_1_COMPLETE.md` - Feature documentation
+- `IMPLEMENTATION_STATUS.md` - Current status
+- `PHASE_1_SUMMARY.md` - This file
+
+**Modified:**
+- `src/orchestrator/types.ts` - Added NoteToFutureSelf, CoordinateAction
+- `src/orchestrator/orchestrator-agent.ts` - Note tracking, coordinate handling, mode switching
+- `src/scenario-worker-class.ts` - Timeout handling (earlier fix)
+- `src/execution-service.ts` - Timeout handling (earlier fix)
+
+---
+
+## Iteration Budget (Max 5 per Step)
+
+**Phase 1 (Current):**
+```
+Iterations 1-3: Playwright selectors (3 attempts)
+Iterations 4-5: Coordinates (2 attempts)
+```
+
+**Phase 2 (Future - Optimized):**
+```
+Iteration 1:   Playwright selector (1 attempt) - fast path
+Iterations 2-3: Index commands (2 attempts) - reliable fallback
+Iterations 4-5: Coordinates (2 attempts) - last resort
+```
+
+**Benefit of Phase 2:**
+- Most scenarios finish in iteration 1 (fast!)
+- Complex scenarios use iterations 2-3 (index system)
+- Only extreme cases reach iterations 4-5 (coordinates)
+
+---
+
+## Ready to Test!
+
+**Current version** (runner-core v0.0.33) is built and ready.
+
+**Test with:**
+1. VS Code extension "Generate Script" on `peoplehr.txt`
+2. Or "Run Test" on any existing smart test
+3. Check logs for note-to-self and coordinate usage
+
+**After validating Phase 1 works well, proceed to Phase 2 for numbered element system.**
+

package/plandocs/PROMPT_OPTIMIZATION_ANALYSIS.md

@@ -0,0 +1,120 @@
+# System Prompt Optimization Analysis
+
+## Current Stats:
+- **System Prompt**: 17,573 chars (346 lines)
+- **With Tool Descriptions**: 19,613 chars (~4,903 tokens)
+- **Cost per call**: ~$0.0007 (gpt-5-mini input tokens)
+
+## Optimization Opportunities:
+
+### 1. **Duplicate Examples** (Save ~30%)
+**Current**: Multiple example sections with ❌/✅ pairs
+- Lines 633-644: Examples section with goto, fill, click examples
+- Lines 621-626: Ambiguous text handling examples  
+- Lines 603-607: DOM snapshot examples
+- Lines 615-619: Selector preference list
+
+**Optimization**: Consolidate into ONE examples section
+**Savings**: ~2,000 chars
+
+### 2. **Verbose Selector Section** (Save ~20%)
+**Current**: Lines 602-644 (42 lines, ~1,800 chars)
+- Lists all selector types with emoji
+- Detailed examples for each
+- Repetitive "Good/Bad" patterns
+
+**Optimization**: Create compact reference table
+```
+SELECTORS (preference order):
+1. getByRole/Label/Placeholder (semantic, stable)
+2. getByText (scope to parent if ambiguous!)  
+3. CSS IDs (avoid auto-generated)
+
+Common mistakes: Missing goto timeout, unscoped getByText, auto-generated IDs
+```
+**Savings**: ~1,200 chars
+
+### 3. **Emoji Overuse** (Save ~5%)
+**Current**: Heavy use of ⚠️, ❌, ✅, 🏆, etc.
+
+**Optimization**: Use sparingly (only for critical warnings)
+**Savings**: ~500 chars
+
+### 4. **Redundant "WHY" Explanations** (Save ~10%)
+**Current**: Multiple "WHY:" sections explaining rationale
+- Line 642-644: WHY semantic selectors
+- Similar explanations scattered throughout
+
+**Optimization**: Remove or consolidate
+**Savings**: ~800 chars
+
+### 5. **Tool Instructions Redundancy** (Save ~10%)
+**Current**: Tools described twice:
+- In tool registry (dynamic)
+- In prompt rules (static)
+
+**Optimization**: Rely more on tool registry descriptions
+**Savings**: ~600 chars
+
+### 6. **Status Rules Repetition** (Save ~5%)
+**Current**: Lines 468-486 - Status rules explained multiple times
+
+**Optimization**: Single concise statement
+**Savings**: ~400 chars
+
+## Proposed Condensed Structure:
+
+```markdown
+# System Prompt (Optimized)
+
+## Agent Role & Tools
+[Tool descriptions from registry]
+
+## Response Format (JSON)
+{required fields} - minimal format, no extensive comments
+
+## Core Rules (Prioritized)
+1. Status decisions (complete/continue/stuck)
+2. Selector strategy (semantic > text > CSS)  
+3. Common errors (goto timeout, strict mode, auto-IDs)
+4. When to use tools vs commands
+5. Note to future self usage
+
+## Examples (Consolidated)
+- Navigation: goto with 30s timeout
+- Selectors: Scoped getByText, semantic selectors
+- Coordinates: When and how
+
+## Advanced Features
+- Blocker detection
+- Step re-evaluation
+- Coordinate fallback
+```
+
+## Total Potential Savings:
+
+- **Before**: 17,573 chars (~4,393 tokens)
+- **After**: ~12,000 chars (~3,000 tokens)  
+- **Reduction**: ~32% reduction in system prompt
+- **Cost savings**: ~$0.0002 per call (~30% per call)
+- **Overall impact**: With 7 tasks using gpt-4o-mini, only 4 tasks will benefit
+- **Est. total savings**: ~5-8% additional cost reduction
+
+## Recommendation:
+
+**Optimize if:**
+- You're seeing consistent 500 errors (less likely now with retry)
+- Want to maximize caching efficiency
+- Running high-volume scenarios (1000+ per day)
+
+**Skip if:**
+- Current cost is acceptable
+- Prompt clarity is more important than 5-8% savings
+- Risk of quality degradation concerns you
+
+## Action Items (if optimizing):
+
+1. ✅ Keep: Critical decision logic, JSON format, coordinate mode
+2. ⚠️ Condense: Selector examples, error responses, WHY sections
+3. ❌ Remove: Duplicate examples, excessive emojis, redundant explanations
+

package/plandocs/PROMPT_SANITY_CHECK.md

@@ -0,0 +1,120 @@
+# Prompt Sanity Check - Runner-Core v0.0.33
+
+## ✅ STRENGTHS
+
+### System Prompt (`buildSystemPrompt`)
+- ✅ Required fields clearly marked at top (status, reasoning, statusReasoning)
+- ✅ Comprehensive JSON format with examples
+- ✅ Clear status decision rules
+- ✅ Good blocker detection guidance
+- ✅ Semantic selector preference clearly explained with examples
+- ✅ Tool vs command distinction is clear
+- ✅ Coordinate fallback documented
+
+### User Prompt (`buildUserPrompt`)
+- ✅ Static content first (cache-optimized)
+- ✅ Dynamic content last (current state, page info)
+- ✅ Notes from previous iteration shown prominently
+- ✅ Clear warnings for consecutive failures
+- ✅ Coordinate mode trigger clear
+
+## ⚠️ ISSUES FOUND
+
+### 1. **Duplication/Redundancy**
+- ❌ "Use semantic selectors" mentioned in:
+  - System prompt (line ~605: "SELECTOR PREFERENCE")
+  - User prompt (line ~860: "SELECTOR STRATEGY")
+  - **FIX**: Remove from user prompt, keep in system prompt only
+
+### 2. **Length Concerns**
+- ⚠️ System prompt is ~325 lines (very long)
+- ⚠️ May cause LLM to miss critical details in the middle
+- **SUGGESTION**: Consider breaking into sections or condensing
+
+### 3. **Conflicting Guidance**
+- ⚠️ Line ~469: "stuck: Tried 3+ iterations"
+  - But coordinate mode triggers at 3 failures (line ~904)
+  - **FIX**: Clarify: stuck = 5 attempts total (3 regular + 2 coordinate)
+
+### 4. **Unclear Iteration Count**
+- ❌ Line ~714: "When iteration count reaches 4+"
+- ❌ Line ~748: "iteration 4+"
+- ✅ But code triggers at 3 failures
+- **FIX**: Update prompt to say "iteration 4+" (0,1,2 = 3 failures, next is #3 which is 4th iteration)
+
+### 5. **Missing Information**
+- ❌ Max iterations per step not mentioned (code has 5)
+- **FIX**: Add to system prompt: "MAX 5 iterations per step"
+
+### 6. **Verbosity**
+- ⚠️ Examples section (lines ~617-628) is great but long
+- ⚠️ Multiple emoji warnings (⚠️⚠️⚠️) can be reduced to single ⚠️
+- **SUGGESTION**: Keep examples, reduce emoji spam
+
+## 🔧 RECOMMENDED FIXES
+
+### Priority 1 (Critical):
+1. Remove duplicate selector strategy from user prompt
+2. Clarify max iterations (5 total)
+3. Fix coordinate mode iteration number (4th iteration = after 3 failures)
+
+### Priority 2 (Nice to have):
+4. Condense system prompt if possible (target: 250 lines)
+5. Reduce emoji overuse
+6. Add section headers in system prompt for clarity
+
+## 📊 PROMPT STRUCTURE ANALYSIS
+
+### System Prompt Sections:
+1. Introduction (1 line)
+2. Tool descriptions (dynamic, from registry)
+3. JSON format (40 lines) ✅
+4. Status rules (15 lines) ✅
+5. Step re-evaluation (20 lines) ✅
+6. Blocker detection (25 lines) ✅
+7. Experiences (25 lines) ✅
+8. Critical rules (200 lines) ⚠️ TOO LONG
+9. Coordinate actions (45 lines) ✅
+
+**TOTAL**: ~370 lines (with tool descriptions)
+
+### User Prompt Sections:
+1. Static instructions (20 lines) - **Cache-friendly** ✅
+2. Dynamic context marker (1 line) ✅
+3. Notes from previous iteration (5 lines) ✅
+4. Warnings for failures (15 lines) ✅
+5. Coordinate mode trigger (8 lines) ✅
+6. Current step goal (10 lines) ✅
+7. Page state (50-100 lines, variable) ✅
+8. Recent steps (20-50 lines, variable) ✅
+9. Experiences (10 lines) ✅
+
+**TOTAL**: ~140-200 lines per call
+
+## 🎯 RECOMMENDATION SUMMARY
+
+**Keep as-is:**
+- JSON structure
+- Semantic selector examples
+- Blocker detection
+- Note to future self
+- Coordinate fallback
+- Cache optimization
+
+**Fix:**
+- Remove selector duplication in user prompt
+- Clarify iteration counts
+- Add max iteration limit
+- Reduce emoji spam
+
+**Consider:**
+- Condensing "Critical Rules" section (currently 200 lines)
+- Moving some examples to external docs
+- Breaking long sections with clear headers
+
+## Overall Assessment: **8/10**
+- Prompts are comprehensive and well-structured
+- Main issues are length and minor redundancies
+- Cache optimization is excellent
+- A few clarity fixes needed for iteration counts
+

package/plandocs/SESSION_SUMMARY_v0.0.33.md

@@ -0,0 +1,151 @@
+# Runner-Core v0.0.33 - Session Summary
+
+## Date: October 15, 2025
+
+## Major Accomplishments:
+
+### 1. ✅ **Coordinate Fallback System** (Phase 1 Complete)
+- Percentage-based coordinates (0-100%, 3 decimal precision)
+- Activates after 3 selector failures
+- 2 coordinate attempts before giving up
+- Resolution-independent positioning
+
+### 2. ✅ **Note to Future Self** (Tactical Memory)
+- Free-form notes persist across iterations AND steps
+- Enables strategic planning across agent decisions
+- Helps maintain context: "Tried X, will try Y next"
+
+### 3. ✅ **Visual Verification Tool** (NEW)
+- `verify_action_result` - Before/after screenshot comparison
+- Agent-callable (decides when to use)
+- JPEG 60% quality (85-90% smaller than PNG)
+- Multi-image LLM interface support
+
+### 4. ✅ **Critical Bug Fixes**
+- **Coordinate mode never activated**: Changed forced stuck from >= 3 to >= 5 failures
+- **Missing required fields**: Made parser flexible (accepts reasoning OR statusReasoning)
+- **Navigation timeouts**: Added 30s timeout guidance for page.goto()
+- **Strict mode violations**: Added scoping guidance (locator('#parent').getByText())
+
+### 5. ✅ **Prompt Optimizations**
+- **59% reduction**: 17,573 chars → 7,287 chars in system prompt
+- **Cache-optimized**: Static content first, dynamic last
+- **Cost savings**: ~40% overall with model tiering
+- **Focused on cognition**: Removed bloat, kept decision-making guidance
+
+### 6. ✅ **Model Optimization**
+- **gpt-5-mini**: Complex tasks (4 operations)
+  - Command generation
+  - Goal completion checks
+  - Repair suggestions
+  - Agent orchestration
+- **gpt-4o-mini**: Simple tasks (7 operations)
+  - Scenario breakdown
+  - Screenshot need assessment
+  - Repair confidence
+  - Test name generation
+  - Hashtag generation
+  - Script parsing
+  - Final script merging
+- **Est. 25-30% cost reduction**
+
+### 7. ✅ **Code Cleanup**
+- Removed V1 SmartTestRunnerCore (V2 is stable)
+- Removed backup files (.bak, .tmp)
+- Consolidated types into V2
+- Removed PeopleHR-specific examples from prompts
+
+### 8. ✅ **Enhanced Logging**
+- Prompt length metrics (chars + estimated tokens)
+- Full LLM response on parsing errors
+- Field presence diagnostics
+- Retry logging for 500 errors
+
+### 9. ✅ **Retry Logic**
+- Automatic retry for OpenAI 500 errors
+- Exponential backoff (1s, 2s, 4s)
+- Up to 3 attempts before failing
+
+### 10. ✅ **Headed Mode for Local Testing**
+- All browser instances use headed: false → headed: false for local dev
+- Visual debugging enabled
+
+## Files Modified:
+
+### Runner-Core:
+1. `src/orchestrator/orchestrator-agent.ts` - Main agent logic
+2. `src/orchestrator/types.ts` - NoteToFutureSelf, CoordinateAction
+3. `src/utils/coordinate-converter.ts` - NEW - Coordinate to Playwright conversion
+4. `src/orchestrator/tools/verify-action-result.ts` - NEW - Visual verification tool
+5. `src/llm-provider.ts` - Added LabeledImage, multi-image support
+6. `src/llm-facade.ts` - Model optimization
+7. `src/model-constants.ts` - Added DEFAULT_SIMPLER_MODEL
+8. `src/scenario-worker-class.ts` - Tool registration
+9. `src/orchestrator/index.ts` - Exports
+10. `src/orchestrator/tools/index.ts` - Tool exports
+
+### Scriptservice:
+1. `providers/scriptservice-llm-provider.ts` - Multi-image handling, retry logic
+2. `smart-test-runner-core-v2.ts` - Type definitions, V1 removal
+3. `smart-test-execution-handler.ts` - V1 removal
+4. `workers/test-based-explorer.ts` - V1 removal
+5. `script-generation-handlers.ts` - Headed mode
+6. `script-generation/script-generation-service.ts` - Headed mode
+7. `smart-test-execution-handler.ts` - Headed mode
+
+### Documentation:
+1. `WHATS_NEW_v0.0.33.md`
+2. `PHASE_1_COMPLETE.md`
+3. `PHASE_1_SUMMARY.md`
+4. `IMPLEMENTATION_STATUS.md`
+5. `VISUAL_AGENT_EVOLUTION_PLAN.md`
+6. `PROMPT_SANITY_CHECK.md`
+7. `PROMPT_OPTIMIZATION_ANALYSIS.md`
+8. `COORDINATE_MODE_DIAGNOSIS.md`
+9. `BEFORE_AFTER_VERIFICATION.md`
+10. `TROUBLESHOOTING_SESSION.md`
+
+## Live Test Status:
+
+**Job**: `71b88c60-52f5-4343-aef8-c44ebb07f3e9`
+**Status**: Running (check browser + logs)
+**Watch For**:
+- Step 5 (Employee Information) - Previously problematic
+- Coordinate mode activation
+- verify_action_result tool usage
+- Overall completion
+
+## Key Metrics:
+
+**Cost Optimization:**
+- Prompt size: 59% reduction
+- Model tiering: 7/11 tasks on cheaper model
+- JPEG compression: 85-90% smaller screenshots
+- **Total savings: ~40% cost reduction**
+
+## Next Steps After Test Completes:
+
+1. Check if Step 5 completes successfully
+2. Verify coordinate mode activated if needed
+3. Check if verify_action_result tool was used
+4. Analyze any remaining failures
+5. Iterate on prompts/logic based on results
+
+## Known Issues to Monitor:
+
+1. **Step 5 False Positive**: Clicking menu item vs navigating to page
+2. **Coordinate Loop**: Agent not knowing when coordinate clicks succeed
+3. **Vision verification usage**: Will agent call it proactively?
+
+## Success Criteria:
+
+✅ All 7 steps complete
+✅ Coordinate fallback used when selectors fail
+✅ Visual verification validates goal achievement
+✅ No infinite loops or stuck states
+✅ Generated script is accurate
+
+---
+
+**Check your browser window and /tmp/scriptservice-test.log for live execution!**
+

package/plandocs/TROUBLESHOOTING_SESSION.md

@@ -0,0 +1,72 @@
+# Troubleshooting Session: All Modules Icon Click Failure
+
+## Objective:
+Understand why the orchestrator agent gets stuck on "Click on the all Modules menu item (top menu icon)" while manual Playwright MCP navigation succeeded.
+
+## What I Need to See:
+
+### 1. Full Agent Logs for the Failing Step
+Please provide the complete logs showing:
+- What iteration attempts were made (iteration 1, 2, 3...)
+- What selectors the agent tried each time
+- What errors it encountered
+- What the DOM snapshot showed
+- Whether it took screenshots
+- What notes it left to future self
+
+### 2. The DOM Context It Saw
+- Interactive elements list
+- ARIA tree snapshot
+- Whether the hamburger icon was visible in the list
+
+## What Worked (My Manual MCP Session):
+
+From earlier successful navigation:
+```
+✅ Step 1: Clicked hamburger menu
+   Selector: #sidebar-toggle > span > svg
+   
+✅ Step 2: Clicked "Core HR"  
+   Selector: getByText('Core HR')
+   
+✅ Step 3: Clicked "Employee Information"
+   Selector: getByText('Employee Information')
+```
+
+## Hypothesis of Why Agent Fails:
+
+### Possible Issue 1: Wrong Selector Strategy
+- Agent might be trying: `getByText('All Modules')` (strict mode violation)
+- Or: `#MenuToggle` (wrong ID)
+- Or: `#sidebar-toggle-menu` (doesn't exist)
+- Instead of: `#sidebar-toggle > span > svg` (actual selector)
+
+### Possible Issue 2: Missing Icon Detection
+- Hamburger icons are often SVG elements without accessible text
+- Agent might not recognize this pattern
+- Prompt doesn't explicitly guide on icon/SVG selector strategy
+
+### Possible Issue 3: DOM List Incomplete
+- Interactive elements might not include the SVG icon
+- If icon isn't in the list, agent won't know it exists
+- Need to check if `getEnhancedPageInfo` captures SVG icons
+
+### Possible Issue 4: Ambiguous Text
+- "All Modules" might appear in multiple places (menu button + modal title)
+- Agent tries `getByText('All Modules')` → strict mode violation
+- Should scope to parent: `locator('#sidebar-toggle').getByText('All Modules')`
+
+## Next Steps:
+
+1. **Get full logs** from your failing run
+2. **Compare** what agent saw vs what I saw  
+3. **Identify** the gap (prompt, DOM extraction, or selector logic)
+4. **Plan fixes**:
+   - Prompt improvements (icon/SVG guidance)
+   - DOM extraction improvements (ensure icons are captured)
+   - Selector strategy improvements (parent scoping for icons)
+   - Example-based learning (hamburger menu pattern)
+
+## Waiting For:
+Please paste the full logs from the failing step showing all iteration attempts and what the agent tried.
+