testchimp-runner-core 0.0.32 → 0.0.34

package/plandocs/PHASE_1_COMPLETE.md

@@ -0,0 +1,165 @@
+# Phase 1 Implementation - COMPLETE ✅
+
+## Version: runner-core v0.0.33
+
+## What's Been Implemented
+
+### 1. Free-Form "Note to Future Self" 
+**Purpose:** Tactical memory - agent leaves notes that persist across iterations AND steps.
+
+**Type:**
+```typescript
+interface NoteToFutureSelf {
+  fromIteration: number;
+  content: string;  // FREE-FORM - agent writes whatever it wants
+}
+```
+
+**How it works:**
+- Agent includes `"noteToFutureSelf": "..."` in response
+- System stores it in `memory.latestNote` (persists across steps!)
+- Passed to next iteration AND next step
+- Displayed prominently at top of prompt
+- Agent reads it FIRST before making decision
+
+**Scope:** Entire scenario journey (not just current step)
+
+**Example notes:**
+
+*Iteration-specific:*
+- "Tried #sidebar-toggle, failed with 'not clickable'. Will try child SVG element next."
+
+*Step-spanning:*
+- "This app has slow-loading modals. Always wait 2s after page load before clicking."
+- "Cookie consent appears on every page. Check for and dismiss it first."
+- "Sidebar only visible on desktop viewport (>1024px width)."
+
+### 2. Percentage-Based Coordinate Fallback
+**Purpose:** Last-resort mechanism when selector generation repeatedly fails.
+
+**Type:**
+```typescript
+interface CoordinateAction {
+  type: 'coordinate';
+  action: 'click' | 'doubleClick' | 'rightClick' | 'hover' | 'drag' | 'fill' | 'scroll';
+  xPercent: number;  // 0-100, 3 decimal precision
+  yPercent: number;
+  toXPercent?: number;  // For drag
+  toYPercent?: number;
+  value?: string;  // For fill
+  scrollAmount?: number;  // For scroll
+}
+```
+
+**How it works:**
+- LLM outputs percentages: `{xPercent: 15.755, yPercent: 8.500}`
+- CoordinateConverter converts to pixels: `15.755% → 252px`
+- Generates Playwright command: `await page.mouse.click(252, 68);`
+
+**Supported actions:**
+- click, doubleClick, rightClick, hover
+- fill (clicks then types value)
+- drag (from x%,y% to toX%,toY%)
+- scroll (at position, by amount)
+
+### 3. Two-Tier Auto-Escalation
+**Trigger:** Code-controlled (not LLM-decided)
+
+```
+Tier 1 (iterations 1-3): Playwright Selector Mode
+├─ Normal buildSystemPrompt()
+├─ Agent generates: await page.getByRole(...).click()
+├─ Leaves noteToFutureSelf for continuity
+└─ 3 attempts, then escalate
+
+Tier 2 (iterations 4-5): Coordinate Mode
+├─ Auto-activates when consecutiveFailures >= 3
+├─ Uses buildCoordinateSystemPrompt()
+├─ Agent outputs: {xPercent: 15.755, yPercent: 8.500}
+├─ CoordinateConverter → mouse.click(x, y)
+└─ 2 attempts max, then give up
+
+Total: Maximum 5 iterations per step
+```
+
+### 4. Precision & Accuracy
+- **3 decimal precision** for coordinates (~1px accuracy on most screens)
+- **Resolution-independent** - works on any viewport size
+- **Percentage reference:**
+  - Top-left: (0, 0)
+  - Top-right: (100, 0)
+  - Center: (50, 50)
+  - Bottom-right: (100, 100)
+
+## Files Modified
+
+1. **orchestrator/types.ts**
+   - Added `NoteToFutureSelf` interface
+   - Added `CoordinateAction` interface
+   - Updated `AgentDecision` with new fields
+   - Updated `AgentContext` with noteFromPreviousIteration
+
+2. **orchestrator/orchestrator-agent.ts**
+   - Added note tracking in executeStep()
+   - Added coordinate action execution
+   - Added buildCoordinateSystemPrompt()
+   - Updated buildUserPrompt() to display notes
+   - Added mode switching in callAgent()
+   - Updated response format documentation
+
+3. **utils/coordinate-converter.ts** (NEW)
+   - percentToPixels() - Convert % to pixels
+   - getViewportSize() - Get current viewport dimensions
+   - generateCommands() - Create Playwright commands from percentages
+   - executeAction() - Direct execution helper
+
+4. **scenario-worker-class.ts** (Earlier fix)
+   - Smart timeout handling for waitForLoadState
+
+5. **execution-service.ts** (Earlier fix)
+   - Smart timeout handling for navigation commands
+
+## How to Use
+
+**No code changes needed!** The features activate automatically:
+
+1. **Note to self:** Agent can optionally include `noteToFutureSelf` in any iteration
+2. **Coordinates:** Auto-activate at iteration 4 if selectors keep failing
+
+## Testing Phase 1
+
+To validate the implementation:
+
+1. **Run PeopleHR scenario** (previously failed on hamburger menu)
+   - Should now succeed with note guidance
+   - May use coordinates if SVG selector still fails
+
+2. **Check logs for:**
+   - `📝 Note to self: ...` (agent leaving tactical notes)
+   - `🎯 COORDINATE MODE ACTIVATED` (tier 2 triggered)
+   - `🎯 Coordinate Action: click at (X%, Y%)` (using fallback)
+
+3. **Expected improvements:**
+   - 20-30% fewer iterations per step (thanks to notes)
+   - < 5% scenarios need coordinate fallback
+   - Coordinates work when everything else fails
+
+## Phase 2 Preview (Not Yet Implemented)
+
+When Phase 2 is added, it will become a **three-tier** system:
+- Tier 1 (iterations 1-2): Playwright selectors
+- Tier 2 (iterations 3-4): Numbered elements (CLICK[3])
+- Tier 3 (iterations 5+): Percentage coordinates
+
+Phase 2 adds visual markers [1], [2], [3] on elements with structured commands.
+
+---
+
+## Status: ✅ READY FOR TESTING
+
+Runner-core v0.0.33 is built and ready. Test it with:
+- VS Code extension "Run Test" on peoplehr-corrected.smart.spec.ts
+- Or generate new script from peoplehr.txt scenario
+
+**Next:** Validate Phase 1 works before starting Phase 2.
+

package/plandocs/PHASE_1_SUMMARY.md

@@ -0,0 +1,184 @@
+# Phase 1 Complete - Summary & Testing Guide
+
+## Version: runner-core v0.0.33 ✅
+
+---
+
+## Implementation Complete
+
+### What's New:
+
+1. **📝 Note to Future Self**
+   - Free-form tactical memory between iterations
+   - Agent writes: "Tried X, failed. Will try Y next."
+   - Prevents repeated mistakes
+
+2. **🎯 Percentage-Based Coordinates**  
+   - Last-resort fallback (3-decimal precision)
+   - Resolution-independent (works any viewport size)
+   - Supports: click, fill, drag, hover, scroll
+
+3. **⚡ Optimized Iteration Limits**
+   - Max 5 iterations per step (down from 8)
+   - 2 coordinate attempts max (coordinates work or they don't)
+   - Faster feedback on stuck scenarios
+
+---
+
+## Current Behavior (Phase 1)
+
+```
+┌─────────────────────────────────────────────────────┐
+│  Iteration 1: Playwright selector                   │
+│    Try: await page.getByRole('button'...).click()   │
+│    Note: "If this fails, try #id selector"          │
+│                                                      │
+│  Iteration 2: Playwright selector                   │
+│    Read note from iteration 1                       │
+│    Try: await page.locator('#sidebar-toggle').click()│
+│    Note: "If this fails, try SVG child"             │
+│                                                      │
+│  Iteration 3: Playwright selector                   │
+│    Read note from iteration 2                       │
+│    Try: await page.locator('#sidebar-toggle svg')   │
+│    → Fails again                                     │
+│                                                      │
+│  🎯 COORDINATE MODE ACTIVATED 🎯                     │
+│                                                      │
+│  Iteration 4: Coordinate action                     │
+│    Agent outputs: {xPercent: 5.500, yPercent: 8.250}│
+│    Execute: page.mouse.click(88, 66)                │
+│    → Success!                                        │
+│                                                      │
+│  OR if fails...                                      │
+│                                                      │
+│  Iteration 5: Coordinate action (2nd attempt)       │
+│    Try slightly adjusted coordinates                 │
+│    → If fails: GIVE UP (stuck)                      │
+│                                                      │
+│  Total: Max 5 iterations                            │
+└─────────────────────────────────────────────────────┘
+```
+
+---
+
+## Testing Phase 1
+
+### Test 1: PeopleHR Scenario (Previously Failed)
+
+**Expected outcome:**
+- Iteration 1-2: Try text/ID selectors → fail
+- Iteration 3: Note says "try SVG child" → succeeds!
+- OR Iteration 4: Coordinates → succeeds!
+
+**Run:**
+```bash
+# Via VS extension: "Generate Script" on peoplehr.txt
+# Or "Run Test" on peoplehr-corrected.smart.spec.ts
+```
+
+**Look for in logs:**
+```
+📝 Note to self: ...
+🎯 COORDINATE MODE ACTIVATED
+🎯 Coordinate Action (attempt 1/2): click at (5.500%, 8.250%)
+```
+
+### Test 2: Simple Scenario (Should Still Be Fast)
+
+Create test: `simple-login.txt`
+```
+- go to https://example.com/login
+- fill username with "alice"
+- fill password with "password123"
+- click login button
+```
+
+**Expected:**
+- Each step: 1 iteration (Tier 1 success)
+- No coordinates needed
+- Fast execution
+
+### Test 3: Coordinate Fallback
+
+**Deliberately difficult scenario:**
+```
+- go to https://some-app-with-shadow-dom.com
+- click on custom web component icon
+```
+
+**Expected:**
+- Iterations 1-3: Selectors fail
+- Iteration 4: Coordinates succeed
+- Generated script contains: `await page.mouse.click(x, y);`
+
+---
+
+## Expected Improvements
+
+### Metrics to Track:
+
+1. **Iteration Efficiency**
+   - Before: ~4 average iterations per step
+   - After: ~2.5 average iterations per step (30-40% reduction)
+
+2. **Success Rate**
+   - Before: Stuck on complex UIs (hamburgers, icons, shadow DOM)
+   - After: Coordinates provide escape hatch
+
+3. **Coordinate Usage**
+   - Target: < 10% of scenarios use coordinates
+   - Most scenarios still succeed with selectors
+
+---
+
+## Files Changed
+
+**New:**
+- `src/utils/coordinate-converter.ts` - Percentage conversion utility
+- `VISUAL_AGENT_EVOLUTION_PLAN.md` - Complete plan
+- `PHASE_1_COMPLETE.md` - Feature documentation
+- `IMPLEMENTATION_STATUS.md` - Current status
+- `PHASE_1_SUMMARY.md` - This file
+
+**Modified:**
+- `src/orchestrator/types.ts` - Added NoteToFutureSelf, CoordinateAction
+- `src/orchestrator/orchestrator-agent.ts` - Note tracking, coordinate handling, mode switching
+- `src/scenario-worker-class.ts` - Timeout handling (earlier fix)
+- `src/execution-service.ts` - Timeout handling (earlier fix)
+
+---
+
+## Iteration Budget (Max 5 per Step)
+
+**Phase 1 (Current):**
+```
+Iterations 1-3: Playwright selectors (3 attempts)
+Iterations 4-5: Coordinates (2 attempts)
+```
+
+**Phase 2 (Future - Optimized):**
+```
+Iteration 1:   Playwright selector (1 attempt) - fast path
+Iterations 2-3: Index commands (2 attempts) - reliable fallback
+Iterations 4-5: Coordinates (2 attempts) - last resort
+```
+
+**Benefit of Phase 2:**
+- Most scenarios finish in iteration 1 (fast!)
+- Complex scenarios use iterations 2-3 (index system)
+- Only extreme cases reach iterations 4-5 (coordinates)
+
+---
+
+## Ready to Test!
+
+**Current version** (runner-core v0.0.33) is built and ready.
+
+**Test with:**
+1. VS Code extension "Generate Script" on `peoplehr.txt`
+2. Or "Run Test" on any existing smart test
+3. Check logs for note-to-self and coordinate usage
+
+**After validating Phase 1 works well, proceed to Phase 2 for numbered element system.**
+

package/plandocs/PROMPT_OPTIMIZATION_ANALYSIS.md

@@ -0,0 +1,120 @@
+# System Prompt Optimization Analysis
+
+## Current Stats:
+- **System Prompt**: 17,573 chars (346 lines)
+- **With Tool Descriptions**: 19,613 chars (~4,903 tokens)
+- **Cost per call**: ~$0.0007 (gpt-5-mini input tokens)
+
+## Optimization Opportunities:
+
+### 1. **Duplicate Examples** (Save ~30%)
+**Current**: Multiple example sections with ❌/✅ pairs
+- Lines 633-644: Examples section with goto, fill, click examples
+- Lines 621-626: Ambiguous text handling examples  
+- Lines 603-607: DOM snapshot examples
+- Lines 615-619: Selector preference list
+
+**Optimization**: Consolidate into ONE examples section
+**Savings**: ~2,000 chars
+
+### 2. **Verbose Selector Section** (Save ~20%)
+**Current**: Lines 602-644 (42 lines, ~1,800 chars)
+- Lists all selector types with emoji
+- Detailed examples for each
+- Repetitive "Good/Bad" patterns
+
+**Optimization**: Create compact reference table
+```
+SELECTORS (preference order):
+1. getByRole/Label/Placeholder (semantic, stable)
+2. getByText (scope to parent if ambiguous!)  
+3. CSS IDs (avoid auto-generated)
+
+Common mistakes: Missing goto timeout, unscoped getByText, auto-generated IDs
+```
+**Savings**: ~1,200 chars
+
+### 3. **Emoji Overuse** (Save ~5%)
+**Current**: Heavy use of ⚠️, ❌, ✅, 🏆, etc.
+
+**Optimization**: Use sparingly (only for critical warnings)
+**Savings**: ~500 chars
+
+### 4. **Redundant "WHY" Explanations** (Save ~10%)
+**Current**: Multiple "WHY:" sections explaining rationale
+- Line 642-644: WHY semantic selectors
+- Similar explanations scattered throughout
+
+**Optimization**: Remove or consolidate
+**Savings**: ~800 chars
+
+### 5. **Tool Instructions Redundancy** (Save ~10%)
+**Current**: Tools described twice:
+- In tool registry (dynamic)
+- In prompt rules (static)
+
+**Optimization**: Rely more on tool registry descriptions
+**Savings**: ~600 chars
+
+### 6. **Status Rules Repetition** (Save ~5%)
+**Current**: Lines 468-486 - Status rules explained multiple times
+
+**Optimization**: Single concise statement
+**Savings**: ~400 chars
+
+## Proposed Condensed Structure:
+
+```markdown
+# System Prompt (Optimized)
+
+## Agent Role & Tools
+[Tool descriptions from registry]
+
+## Response Format (JSON)
+{required fields} - minimal format, no extensive comments
+
+## Core Rules (Prioritized)
+1. Status decisions (complete/continue/stuck)
+2. Selector strategy (semantic > text > CSS)  
+3. Common errors (goto timeout, strict mode, auto-IDs)
+4. When to use tools vs commands
+5. Note to future self usage
+
+## Examples (Consolidated)
+- Navigation: goto with 30s timeout
+- Selectors: Scoped getByText, semantic selectors
+- Coordinates: When and how
+
+## Advanced Features
+- Blocker detection
+- Step re-evaluation
+- Coordinate fallback
+```
+
+## Total Potential Savings:
+
+- **Before**: 17,573 chars (~4,393 tokens)
+- **After**: ~12,000 chars (~3,000 tokens)  
+- **Reduction**: ~32% reduction in system prompt
+- **Cost savings**: ~$0.0002 per call (~30% per call)
+- **Overall impact**: With 7 tasks using gpt-4o-mini, only 4 tasks will benefit
+- **Est. total savings**: ~5-8% additional cost reduction
+
+## Recommendation:
+
+**Optimize if:**
+- You're seeing consistent 500 errors (less likely now with retry)
+- Want to maximize caching efficiency
+- Running high-volume scenarios (1000+ per day)
+
+**Skip if:**
+- Current cost is acceptable
+- Prompt clarity is more important than 5-8% savings
+- Risk of quality degradation concerns you
+
+## Action Items (if optimizing):
+
+1. ✅ Keep: Critical decision logic, JSON format, coordinate mode
+2. ⚠️ Condense: Selector examples, error responses, WHY sections
+3. ❌ Remove: Duplicate examples, excessive emojis, redundant explanations
+

package/plandocs/PROMPT_SANITY_CHECK.md

@@ -0,0 +1,120 @@
+# Prompt Sanity Check - Runner-Core v0.0.33
+
+## ✅ STRENGTHS
+
+### System Prompt (`buildSystemPrompt`)
+- ✅ Required fields clearly marked at top (status, reasoning, statusReasoning)
+- ✅ Comprehensive JSON format with examples
+- ✅ Clear status decision rules
+- ✅ Good blocker detection guidance
+- ✅ Semantic selector preference clearly explained with examples
+- ✅ Tool vs command distinction is clear
+- ✅ Coordinate fallback documented
+
+### User Prompt (`buildUserPrompt`)
+- ✅ Static content first (cache-optimized)
+- ✅ Dynamic content last (current state, page info)
+- ✅ Notes from previous iteration shown prominently
+- ✅ Clear warnings for consecutive failures
+- ✅ Coordinate mode trigger clear
+
+## ⚠️ ISSUES FOUND
+
+### 1. **Duplication/Redundancy**
+- ❌ "Use semantic selectors" mentioned in:
+  - System prompt (line ~605: "SELECTOR PREFERENCE")
+  - User prompt (line ~860: "SELECTOR STRATEGY")
+  - **FIX**: Remove from user prompt, keep in system prompt only
+
+### 2. **Length Concerns**
+- ⚠️ System prompt is ~325 lines (very long)
+- ⚠️ May cause LLM to miss critical details in the middle
+- **SUGGESTION**: Consider breaking into sections or condensing
+
+### 3. **Conflicting Guidance**
+- ⚠️ Line ~469: "stuck: Tried 3+ iterations"
+  - But coordinate mode triggers at 3 failures (line ~904)
+  - **FIX**: Clarify: stuck = 5 attempts total (3 regular + 2 coordinate)
+
+### 4. **Unclear Iteration Count**
+- ❌ Line ~714: "When iteration count reaches 4+"
+- ❌ Line ~748: "iteration 4+"
+- ✅ But code triggers at 3 failures
+- **FIX**: Update prompt to say "iteration 4+" (0,1,2 = 3 failures, next is #3 which is 4th iteration)
+
+### 5. **Missing Information**
+- ❌ Max iterations per step not mentioned (code has 5)
+- **FIX**: Add to system prompt: "MAX 5 iterations per step"
+
+### 6. **Verbosity**
+- ⚠️ Examples section (lines ~617-628) is great but long
+- ⚠️ Multiple emoji warnings (⚠️⚠️⚠️) can be reduced to single ⚠️
+- **SUGGESTION**: Keep examples, reduce emoji spam
+
+## 🔧 RECOMMENDED FIXES
+
+### Priority 1 (Critical):
+1. Remove duplicate selector strategy from user prompt
+2. Clarify max iterations (5 total)
+3. Fix coordinate mode iteration number (4th iteration = after 3 failures)
+
+### Priority 2 (Nice to have):
+4. Condense system prompt if possible (target: 250 lines)
+5. Reduce emoji overuse
+6. Add section headers in system prompt for clarity
+
+## 📊 PROMPT STRUCTURE ANALYSIS
+
+### System Prompt Sections:
+1. Introduction (1 line)
+2. Tool descriptions (dynamic, from registry)
+3. JSON format (40 lines) ✅
+4. Status rules (15 lines) ✅
+5. Step re-evaluation (20 lines) ✅
+6. Blocker detection (25 lines) ✅
+7. Experiences (25 lines) ✅
+8. Critical rules (200 lines) ⚠️ TOO LONG
+9. Coordinate actions (45 lines) ✅
+
+**TOTAL**: ~370 lines (with tool descriptions)
+
+### User Prompt Sections:
+1. Static instructions (20 lines) - **Cache-friendly** ✅
+2. Dynamic context marker (1 line) ✅
+3. Notes from previous iteration (5 lines) ✅
+4. Warnings for failures (15 lines) ✅
+5. Coordinate mode trigger (8 lines) ✅
+6. Current step goal (10 lines) ✅
+7. Page state (50-100 lines, variable) ✅
+8. Recent steps (20-50 lines, variable) ✅
+9. Experiences (10 lines) ✅
+
+**TOTAL**: ~140-200 lines per call
+
+## 🎯 RECOMMENDATION SUMMARY
+
+**Keep as-is:**
+- JSON structure
+- Semantic selector examples
+- Blocker detection
+- Note to future self
+- Coordinate fallback
+- Cache optimization
+
+**Fix:**
+- Remove selector duplication in user prompt
+- Clarify iteration counts
+- Add max iteration limit
+- Reduce emoji spam
+
+**Consider:**
+- Condensing "Critical Rules" section (currently 200 lines)
+- Moving some examples to external docs
+- Breaking long sections with clear headers
+
+## Overall Assessment: **8/10**
+- Prompts are comprehensive and well-structured
+- Main issues are length and minor redundancies
+- Cache optimization is excellent
+- A few clarity fixes needed for iteration counts
+