testchimp-runner-core 0.0.35 → 0.0.37

package/plandocs/PHASE_1_SUMMARY.md

@@ -1,184 +0,0 @@
-# Phase 1 Complete - Summary & Testing Guide
-
-## Version: runner-core v0.0.33 ✅
-
----
-
-## Implementation Complete
-
-### What's New:
-
-1. **📝 Note to Future Self**
-   - Free-form tactical memory between iterations
-   - Agent writes: "Tried X, failed. Will try Y next."
-   - Prevents repeated mistakes
-
-2. **🎯 Percentage-Based Coordinates**  
-   - Last-resort fallback (3-decimal precision)
-   - Resolution-independent (works any viewport size)
-   - Supports: click, fill, drag, hover, scroll
-
-3. **⚡ Optimized Iteration Limits**
-   - Max 5 iterations per step (down from 8)
-   - 2 coordinate attempts max (coordinates work or they don't)
-   - Faster feedback on stuck scenarios
-
----
-
-## Current Behavior (Phase 1)
-
-```
-┌─────────────────────────────────────────────────────┐
-│  Iteration 1: Playwright selector                   │
-│    Try: await page.getByRole('button'...).click()   │
-│    Note: "If this fails, try #id selector"          │
-│                                                      │
-│  Iteration 2: Playwright selector                   │
-│    Read note from iteration 1                       │
-│    Try: await page.locator('#sidebar-toggle').click()│
-│    Note: "If this fails, try SVG child"             │
-│                                                      │
-│  Iteration 3: Playwright selector                   │
-│    Read note from iteration 2                       │
-│    Try: await page.locator('#sidebar-toggle svg')   │
-│    → Fails again                                     │
-│                                                      │
-│  🎯 COORDINATE MODE ACTIVATED 🎯                     │
-│                                                      │
-│  Iteration 4: Coordinate action                     │
-│    Agent outputs: {xPercent: 5.500, yPercent: 8.250}│
-│    Execute: page.mouse.click(88, 66)                │
-│    → Success!                                        │
-│                                                      │
-│  OR if fails...                                      │
-│                                                      │
-│  Iteration 5: Coordinate action (2nd attempt)       │
-│    Try slightly adjusted coordinates                 │
-│    → If fails: GIVE UP (stuck)                      │
-│                                                      │
-│  Total: Max 5 iterations                            │
-└─────────────────────────────────────────────────────┘
-```
-
----
-
-## Testing Phase 1
-
-### Test 1: PeopleHR Scenario (Previously Failed)
-
-**Expected outcome:**
-- Iteration 1-2: Try text/ID selectors → fail
-- Iteration 3: Note says "try SVG child" → succeeds!
-- OR Iteration 4: Coordinates → succeeds!
-
-**Run:**
-```bash
-# Via VS extension: "Generate Script" on peoplehr.txt
-# Or "Run Test" on peoplehr-corrected.smart.spec.ts
-```
-
-**Look for in logs:**
-```
-📝 Note to self: ...
-🎯 COORDINATE MODE ACTIVATED
-🎯 Coordinate Action (attempt 1/2): click at (5.500%, 8.250%)
-```
-
-### Test 2: Simple Scenario (Should Still Be Fast)
-
-Create test: `simple-login.txt`
-```
-- go to https://example.com/login
-- fill username with "alice"
-- fill password with "password123"
-- click login button
-```
-
-**Expected:**
-- Each step: 1 iteration (Tier 1 success)
-- No coordinates needed
-- Fast execution
-
-### Test 3: Coordinate Fallback
-
-**Deliberately difficult scenario:**
-```
-- go to https://some-app-with-shadow-dom.com
-- click on custom web component icon
-```
-
-**Expected:**
-- Iterations 1-3: Selectors fail
-- Iteration 4: Coordinates succeed
-- Generated script contains: `await page.mouse.click(x, y);`
-
----
-
-## Expected Improvements
-
-### Metrics to Track:
-
-1. **Iteration Efficiency**
-   - Before: ~4 average iterations per step
-   - After: ~2.5 average iterations per step (30-40% reduction)
-
-2. **Success Rate**
-   - Before: Stuck on complex UIs (hamburgers, icons, shadow DOM)
-   - After: Coordinates provide escape hatch
-
-3. **Coordinate Usage**
-   - Target: < 10% of scenarios use coordinates
-   - Most scenarios still succeed with selectors
-
----
-
-## Files Changed
-
-**New:**
-- `src/utils/coordinate-converter.ts` - Percentage conversion utility
-- `VISUAL_AGENT_EVOLUTION_PLAN.md` - Complete plan
-- `PHASE_1_COMPLETE.md` - Feature documentation
-- `IMPLEMENTATION_STATUS.md` - Current status
-- `PHASE_1_SUMMARY.md` - This file
-
-**Modified:**
-- `src/orchestrator/types.ts` - Added NoteToFutureSelf, CoordinateAction
-- `src/orchestrator/orchestrator-agent.ts` - Note tracking, coordinate handling, mode switching
-- `src/scenario-worker-class.ts` - Timeout handling (earlier fix)
-- `src/execution-service.ts` - Timeout handling (earlier fix)
-
----
-
-## Iteration Budget (Max 5 per Step)
-
-**Phase 1 (Current):**
-```
-Iterations 1-3: Playwright selectors (3 attempts)
-Iterations 4-5: Coordinates (2 attempts)
-```
-
-**Phase 2 (Future - Optimized):**
-```
-Iteration 1:   Playwright selector (1 attempt) - fast path
-Iterations 2-3: Index commands (2 attempts) - reliable fallback
-Iterations 4-5: Coordinates (2 attempts) - last resort
-```
-
-**Benefit of Phase 2:**
-- Most scenarios finish in iteration 1 (fast!)
-- Complex scenarios use iterations 2-3 (index system)
-- Only extreme cases reach iterations 4-5 (coordinates)
-
----
-
-## Ready to Test!
-
-**Current version** (runner-core v0.0.33) is built and ready.
-
-**Test with:**
-1. VS Code extension "Generate Script" on `peoplehr.txt`
-2. Or "Run Test" on any existing smart test
-3. Check logs for note-to-self and coordinate usage
-
-**After validating Phase 1 works well, proceed to Phase 2 for numbered element system.**
-

package/plandocs/PLANNING_SESSION_SUMMARY.md

@@ -1,372 +0,0 @@
-# Planning Session Summary: Orchestrator Agent Architecture
-
-## Date: October 11, 2025
-
----
-
-## Final Decisions Made
-
-### 1. ✅ Self-Reflection in MVP
-**Decision**: Include free-form self-reflection with agent-driven loop detection
-- Agent outputs `guidanceForNext` (free-form text) for train of thought continuity
-- Agent signals `detectingLoop: true` when it notices repetition
-- Agent decides when to break own loop, system enforces hard limits as backup
-
-**Rationale**: Valuable for maintaining context across iterations, agent self-corrects
-
-### 2. ✅ No Screenshot Budget
-**Decision**: Screenshots available freely, no artificial limits
-- Corrected token cost: 1-2K tokens (NOT 100K!)
-- For 1920x1080 viewport: ~1,452 tokens (gpt-4.1-mini)
-- Comparable to extra DOM context
-
-**Rationale**: Very affordable, enables liberal vision use throughout journey
-
-### 3. ✅ DOM Limits (Increased for Complex Pages)
-**Decision**: Increased limits in getEnhancedPageInfo to handle complex UIs
-- ARIA tree depth: 4 levels
-- Interactive elements: top 50 (was 12)
-- IDs: top 50 (was 10)
-- Data attributes: top 50 (was 10)
-- Form fields: top 20 (was 8)
-- Page structure: top 10 (was 6)
-- General elements: top 50 (was 15)
-- Text: 30 chars max
-- Result: ~800-1,500 tokens
-
-**Rationale**: Complex pages need more context, still compact with truncation
-
-### 4. ✅ Token Usage Tracking
-**Decision**: Track and report all LLM token usage via callback
-- Interface: `onTokensUsed({inputTokens, outputTokens, includesImage})`
-- Heuristic: 4 characters = 1 token
-- Image tokens: ~1,500 estimate for viewport screenshots
-- Reported via ProgressReporter for analytics
-
-**Rationale**: Cost tracking, optimization, analytics
-
-### 5. ✅ Recovery Tools in MVP
-**Decision**: Include 3 recovery tools for self-unsticking
-- `navigate_back()` - Go back in history
-- `refresh_page()` - Reload page
-- `navigate_to_url({url})` - Navigate to specific URL (with domain validation)
-
-**Rationale**: Agent needs ability to recover from bad states (wrong navigation, stuck page, side effects)
-
-### 6. ✅ Inquisitive Exploration in Phase 2
-**Decision**: Defer exploratory actions to Phase 2, MVP uses workarounds
-- Phase 2 tool: `explore_element({action, selector, purpose})`
-- Actions: hover, click_info, click_menu, focus
-- Safety: State validation, non-consequential only
-- **Screenshot handling**: Immediate analysis via sub-agent call
-  - System takes screenshot after exploration
-  - Calls agent to analyze screenshot
-  - Agent extracts learnings (text)
-  - Only TEXT stored in history, NOT screenshot
-  - Keeps memory lightweight
-- MVP workaround: Use screenshot + DOM analysis + retry
-
-**Rationale**: Safety concerns, complexity, needs battle-testing first
-
-### 7. ✅ Always-Provided Context Structure
-**Decision**: Provide comprehensive context automatically each iteration
-- Overall goal + current goal
-- Current page info (DOM)
-- Recent 6-7 steps
-- Experiences (learnings)
-- Extracted data
-- Self-reflection from previous iteration
-- Journey progress tracking
-
-**Rationale**: Agent needs full situational awareness without repeated tool calls
-
-### 8. ✅ System vs Agent Guardrails
-**Decision**: Clear separation of responsibilities
-- **System enforces**: Iteration limits, tool call limits, command limits
-- **Agent signals**: Stuck, infeasible, detecting loop
-- System has final say, agent provides soft guidance
-
-**Rationale**: Safety (hard limits) + intelligence (agent self-awareness)
-
----
-
-## Architecture Summary
-
-### Core Components
-
-**1. Always-Provided Context** (auto-fetched each iteration)
-```typescript
-{
-  overallGoal, currentStepGoal, stepNumber, totalSteps,
-  currentPageInfo, currentURL,
-  recentSteps (6-7), experiences, extractedData,
-  previousIterationGuidance
-}
-```
-
-**2. Tools** (8 in MVP)
-- **Information**: take_screenshot, recall_history, inspect_page, check_page_ready
-- **Data**: extract_data
-- **Recovery**: navigate_back, refresh_page, navigate_to_url
-
-**3. Agent Decision Output**
-```typescript
-{
-  toolCalls, toolReasoning, needsToolResults,
-  commands, commandReasoning,
-  selfReflection: {guidanceForNext, detectingLoop, loopReasoning},
-  experiences, memoryUpdate,
-  status: 'complete' | 'stuck' | 'infeasible' | 'continue',
-  statusReasoning, reasoning
-}
-```
-
-**4. Sequential Batch Execution**
-- Agent plans batch of commands (max 3-5)
-- System executes one-by-one
-- Stop at first failure
-- Record each individually in history
-
-**5. Comprehensive Logging**
-- Every iteration: goal, reasoning, self-reflection, tools, commands, experiences, status
-- All thoughts visible for debugging
-- Exported via ProgressReporter
-
-**6. Token Usage Tracking**
-- Input + output tokens calculated (4 chars = 1 token)
-- Image tokens estimated (~1,500 for viewport)
-- Reported via `onTokensUsed()` callback
-
----
-
-## MVP vs Phase 2
-
-### MVP Includes:
-- ✅ 8 core tools (info + data + recovery)
-- ✅ Journey memory with experiences
-- ✅ Self-reflection + loop detection
-- ✅ Batch command planning
-- ✅ Self-recovery (navigate back/refresh)
-- ✅ Token tracking
-- ✅ Comprehensive logging
-- ✅ Configurable guardrails
-
-### Phase 2 Adds:
-- Inquisitive exploration (explore_element)
-- Advanced optimizations (caching, adaptive limits)
-- Memory summarization for long journeys
-
----
-
-## Key Metrics
-
-### Token Usage Per Iteration (Estimated)
-```
-System prompt:              500 tokens
-Always-provided context:    1,200-2,000 tokens
-  - Goals & progress:       100
-  - DOM (increased limits): 800-1,500
-  - Recent 6-7 steps:       300-500
-  - Experiences:            100-200
-Self-reflection:            100 tokens
-Tool results (optional):    300-500 tokens
-Screenshot (optional):      1,500 tokens
-
-Total without screenshot:   2,400-3,600 tokens
-Total with screenshot:      3,900-5,100 tokens
-```
-
-### Expected Performance
-- LLM calls/step: 2-4 (vs 4-6 current)
-- Iterations/step: 3-5 (vs 8-12 current)
-- Tool calls/step: 1-3
-- Commands/iteration: 2-3 (batched)
-- Agent learns: 1-2 experiences per step
-
----
-
-## Inquisitive Exploration Design (Phase 2)
-
-### Problem
-Menu items are icon-only, no text/ARIA labels → Agent unsure which to click
-
-### Solution
-Agent investigates non-consequentially, analyzes immediately:
-
-```
-Iteration N:
-  Agent Decision: "Need to hover over icons to see tooltips"
-  Tool: explore_element({action: "hover", selector: "nav button:nth-child(2)"})
-  
-  System:
-    → Hover, wait 500ms
-    → Take screenshot
-    → Call agent (sub-call): "What do you see in this screenshot?"
-  
-  Agent Analysis (sub-call): 
-    → Sees tooltip
-    → Returns: "Tooltip shows 'Dashboard' - this is the Dashboard button"
-  
-  System:
-    → Stores TEXT in history: "Explored button, tooltip confirms Dashboard"
-    → Does NOT store screenshot
-    → Returns to main agent: {success: true, learning: "Tooltip shows Dashboard"}
-  
-  Agent Decision (continues): 
-    → "Great, confirmed it's Dashboard"
-    → Commands: ["page.click('nav button:nth-child(2)')"]
-  
-  System: Execute commands
-```
-
-**Key difference**: Screenshot analyzed WITHIN same iteration, only text stored
-
-### Allowed Actions
-- ✅ hover (show tooltips)
-- ✅ click_info (info icons)
-- ✅ click_menu (expand menus)
-- ✅ focus (see input hints)
-
-### NOT Allowed
-- ❌ Submit forms
-- ❌ Delete/remove
-- ❌ Logout
-- ❌ File uploads
-
-### Safety
-- State validation (URL, modal count)
-- Revert if unexpected navigation
-- Budget: 10 explorations per step
-- Timeout: 2s per exploration
-
-### Why Phase 2
-- Safety risk (needs robust validation)
-- Complexity (screenshot handling, state comparison)
-- MVP workaround: screenshot + DOM + retry
-
----
-
-## Implementation Status
-
-### Completed (During Planning):
-- ✅ Token usage tracking added to interfaces
-- ✅ BackendProxyLLMProvider calculates token usage
-- ✅ LLMFacade prepared for token callback
-- ✅ Progress reporter extended with `onTokensUsed()`
-
-### Ready to Implement:
-1. OrchestratorAgent class
-2. ToolRegistry with 8 tools
-3. Journey memory implementation
-4. Always-provided context builder
-5. Self-reflection structures
-6. Recovery tools (navigate_back, refresh, navigate_to)
-7. Comprehensive logging
-8. Token tracking integration
-
-### Phase 2:
-1. Exploratory actions (explore_element)
-2. State validation logic
-3. Advanced optimizations
-
----
-
-## Documentation Created
-
-1. **MULTI_AGENT_ARCHITECTURE_REVIEW.md**
-   - 8 pitfalls analyzed with mitigations
-   - Phased implementation strategy
-   - Risk analysis and trade-offs
-
-2. **ORCHESTRATOR_IMPLEMENTATION_PLAN.md**
-   - Detailed implementation specs
-   - Code examples
-   - Integration points
-
-3. **ORCHESTRATOR_MVP_SUMMARY.md**
-   - Executive summary
-   - Complete feature list
-   - Inquisitive exploration section
-   - MVP vs Phase 2 breakdown
-
-4. **PLANNING_SESSION_SUMMARY.md** (this document)
-   - All decisions made
-   - Rationales
-   - Implementation status
-
----
-
-## Next Steps
-
-### When Ready to Implement:
-1. Review all 3 architecture documents
-2. Start with MVP (8 tools, no exploration)
-3. Implement in order:
-   - Tool registry + tool implementations
-   - Journey memory structures
-   - OrchestratorAgent loop
-   - Integration with ScenarioWorker
-   - Token tracking integration
-   - Comprehensive logging
-4. Test with real scenarios
-5. Measure metrics vs current approach
-6. Iterate based on findings
-7. Add Phase 2 features when validated
-
----
-
-## Success Criteria (MVP)
-
-### Must Have:
-- [ ] Fewer iterations than current (target: 50% reduction)
-- [ ] Backward compatible (VS Extension & GitHub Runner work)
-- [ ] No infinite loops (guardrails work)
-- [ ] Memory doesn't bloat
-- [ ] Tool extensibility works
-- [ ] Token usage tracked accurately
-
-### Nice to Have:
-- [ ] Fewer LLM calls than current
-- [ ] Better success rate
-- [ ] Faster execution
-
-### Acceptable Trade-offs:
-- ⚠️ Slightly higher token usage per iteration (richer context)
-- ⚠️ Some tool call overhead
-- ⚠️ No exploratory actions in MVP
-
----
-
-## Estimated Timeline
-
-**MVP Implementation**: 2-3 weeks
-- Week 1: Foundation (types, tool registry, tool implementations)
-- Week 2: Orchestrator loop, integration
-- Week 3: Testing, refinement, metrics
-
-**Phase 2 (Exploration)**: 1-2 weeks after MVP validated
-
-**Total**: 3-5 weeks for complete solution
-
----
-
-## Final Architecture Confidence
-
-**✅ Ready to implement** with:
-- All major decisions finalized
-- Trade-offs understood and accepted
-- Risks identified with mitigations
-- Phased approach reduces implementation risk
-- Backward compatibility ensured
-- Comprehensive documentation complete
-
-**Key strengths**:
-- Human-like (memory, learning, reflection, recovery)
-- Extensible (tool registry, dynamic prompts)
-- Safe (system guardrails, agent self-awareness)
-- Transparent (comprehensive logging)
-- Cost-aware (token tracking)
-- Practical (recovery tools, self-unstuck)
-
-**No blockers to proceed.**
-

package/plandocs/PROMPT_OPTIMIZATION_ANALYSIS.md

@@ -1,120 +0,0 @@
-# System Prompt Optimization Analysis
-
-## Current Stats:
-- **System Prompt**: 17,573 chars (346 lines)
-- **With Tool Descriptions**: 19,613 chars (~4,903 tokens)
-- **Cost per call**: ~$0.0007 (gpt-5-mini input tokens)
-
-## Optimization Opportunities:
-
-### 1. **Duplicate Examples** (Save ~30%)
-**Current**: Multiple example sections with ❌/✅ pairs
-- Lines 633-644: Examples section with goto, fill, click examples
-- Lines 621-626: Ambiguous text handling examples  
-- Lines 603-607: DOM snapshot examples
-- Lines 615-619: Selector preference list
-
-**Optimization**: Consolidate into ONE examples section
-**Savings**: ~2,000 chars
-
-### 2. **Verbose Selector Section** (Save ~20%)
-**Current**: Lines 602-644 (42 lines, ~1,800 chars)
-- Lists all selector types with emoji
-- Detailed examples for each
-- Repetitive "Good/Bad" patterns
-
-**Optimization**: Create compact reference table
-```
-SELECTORS (preference order):
-1. getByRole/Label/Placeholder (semantic, stable)
-2. getByText (scope to parent if ambiguous!)  
-3. CSS IDs (avoid auto-generated)
-
-Common mistakes: Missing goto timeout, unscoped getByText, auto-generated IDs
-```
-**Savings**: ~1,200 chars
-
-### 3. **Emoji Overuse** (Save ~5%)
-**Current**: Heavy use of ⚠️, ❌, ✅, 🏆, etc.
-
-**Optimization**: Use sparingly (only for critical warnings)
-**Savings**: ~500 chars
-
-### 4. **Redundant "WHY" Explanations** (Save ~10%)
-**Current**: Multiple "WHY:" sections explaining rationale
-- Line 642-644: WHY semantic selectors
-- Similar explanations scattered throughout
-
-**Optimization**: Remove or consolidate
-**Savings**: ~800 chars
-
-### 5. **Tool Instructions Redundancy** (Save ~10%)
-**Current**: Tools described twice:
-- In tool registry (dynamic)
-- In prompt rules (static)
-
-**Optimization**: Rely more on tool registry descriptions
-**Savings**: ~600 chars
-
-### 6. **Status Rules Repetition** (Save ~5%)
-**Current**: Lines 468-486 - Status rules explained multiple times
-
-**Optimization**: Single concise statement
-**Savings**: ~400 chars
-
-## Proposed Condensed Structure:
-
-```markdown
-# System Prompt (Optimized)
-
-## Agent Role & Tools
-[Tool descriptions from registry]
-
-## Response Format (JSON)
-{required fields} - minimal format, no extensive comments
-
-## Core Rules (Prioritized)
-1. Status decisions (complete/continue/stuck)
-2. Selector strategy (semantic > text > CSS)  
-3. Common errors (goto timeout, strict mode, auto-IDs)
-4. When to use tools vs commands
-5. Note to future self usage
-
-## Examples (Consolidated)
-- Navigation: goto with 30s timeout
-- Selectors: Scoped getByText, semantic selectors
-- Coordinates: When and how
-
-## Advanced Features
-- Blocker detection
-- Step re-evaluation
-- Coordinate fallback
-```
-
-## Total Potential Savings:
-
-- **Before**: 17,573 chars (~4,393 tokens)
-- **After**: ~12,000 chars (~3,000 tokens)  
-- **Reduction**: ~32% reduction in system prompt
-- **Cost savings**: ~$0.0002 per call (~30% per call)
-- **Overall impact**: With 7 tasks using gpt-4o-mini, only 4 tasks will benefit
-- **Est. total savings**: ~5-8% additional cost reduction
-
-## Recommendation:
-
-**Optimize if:**
-- You're seeing consistent 500 errors (less likely now with retry)
-- Want to maximize caching efficiency
-- Running high-volume scenarios (1000+ per day)
-
-**Skip if:**
-- Current cost is acceptable
-- Prompt clarity is more important than 5-8% savings
-- Risk of quality degradation concerns you
-
-## Action Items (if optimizing):
-
-1. ✅ Keep: Critical decision logic, JSON format, coordinate mode
-2. ⚠️ Condense: Selector examples, error responses, WHY sections
-3. ❌ Remove: Duplicate examples, excessive emojis, redundant explanations
-