testchimp-runner-core 0.0.35 → 0.0.36

package/plandocs/SELECTOR_IMPROVEMENTS.md

@@ -1,139 +0,0 @@
-# Selector Preference Improvements
-
-## Summary
-Updated the orchestrator agent to prefer user-friendly, semantic Playwright selectors over auto-generated IDs, following Playwright's official best practices.
-
-## Problem
-The agent was generating commands like:
-```typescript
-await page.fill('#«r3»-form-item', 'alice@example.com')
-await page.fill('#«r4»-form-item', 'TestPass123')
-```
-
-These auto-generated IDs (especially with unicode characters like `«r3»`) are:
-- Not user-friendly or readable
-- Break when component instances change
-- Not maintainable
-- Not following Playwright best practices
-
-## Solution
-Implemented a comprehensive selector preference strategy across three key files:
-
-### 1. Orchestrator Agent Prompt (`orchestrator-agent.ts`)
-
-Added new section **"5b. SELECTOR PREFERENCE"** with explicit guidance:
-
-**Preferred selectors (in order):**
-1. `page.getByRole('role', {name: 'text'})` - Accessible, semantic, resilient
-2. `page.getByLabel('label text')` - Great for form inputs
-3. `page.getByPlaceholder('placeholder')` - Good for inputs without labels
-4. `page.getByText('visible text')` - Clear and readable
-5. `page.getByTestId('test-id')` - Stable if available
-
-**Avoid (last resort only):**
-- CSS selectors with auto-generated IDs: `#r3-form-item`, `#«r3»-form-item`
-- CSS selectors with unicode characters
-- Complex CSS paths
-
-**Examples provided:**
-```typescript
-// ❌ BAD
-await page.fill('#«r3»-form-item', 'alice@example.com')
-
-// ✅ GOOD
-await page.getByLabel('Email').fill('alice@example.com')
-await page.getByRole('textbox', {name: 'Email'}).fill('alice@example.com')
-await page.getByPlaceholder('Enter your email').fill('alice@example.com')
-```
-
-### 2. Page Info Utils (`page-info-utils.ts`)
-
-**Reordered selector generation priority:**
-
-Before:
-1. ID selector (e.g., `#«r3»-form-item`)
-2. data-testid
-3. getByRole
-
-After:
-1. getByLabel (for inputs with associated labels)
-2. getByRole with name (semantic and accessible)
-3. getByPlaceholder (for inputs with placeholders)
-4. getByText (visible text fallback)
-5. getByTestId (stable, explicit)
-6. ID selector - ONLY if stable (filters out auto-generated IDs with unicode or patterns like `rc_`, `:r[0-9]+:`, `__`)
-
-**Enhanced display:**
-- Shows best selector first, with up to 2 alternatives
-- Example: `getByLabel('Email') (or: getByRole('textbox', {name: 'Email'}), getByPlaceholder('Enter your email'))`
-
-### 3. Screenshot Tool (`take-screenshot.ts`)
-
-**Updated vision analysis prompts:**
-- System prompt now emphasizes: "ALWAYS prioritize semantic selectors (getByRole, getByLabel, getByText) over CSS selectors with auto-generated IDs"
-- Analysis task explicitly instructs: "Recommend SEMANTIC SELECTORS FIRST" and "AVOID auto-generated IDs with unicode"
-
-### 4. Updated EXPERIENCES Section
-
-Enhanced examples to capture semantic selector patterns:
-```
-✅ GOOD - App-specific patterns:
-- "Login form fields accessible via getByLabel: 'Email' and 'Password'"
-- "Submit buttons consistently use role=button with text matching action"
-- "Input fields have clear placeholders - prefer getByPlaceholder over IDs"
-
-❌ BAD:
-- Noting auto-generated IDs like #«r3»-form-item (these are unreliable)
-```
-
-## Benefits
-
-1. **More Maintainable**: Semantic selectors are resilient to UI changes
-2. **Self-Documenting**: Code reads like natural language
-3. **Accessibility**: Ensures UI elements are properly accessible
-4. **Best Practices**: Follows Playwright's official recommendations
-5. **User-Friendly**: Generated tests are more readable and easier to understand
-
-## Expected Output
-
-After these changes, the agent will generate:
-
-```typescript
-// Login form example
-await page.getByLabel('Email').fill('alice@example.com')
-await page.getByLabel('Password').fill('TestPass123')
-await page.getByRole('button', {name: 'Sign In'}).click()
-
-// Navigation example
-await page.getByRole('link', {name: 'Dashboard'}).click()
-
-// Form with placeholders
-await page.getByPlaceholder('Search...').fill('test query')
-```
-
-Instead of:
-
-```typescript
-// Old style with auto-generated IDs
-await page.fill('#«r3»-form-item', 'alice@example.com')
-await page.fill('#«r4»-form-item', 'TestPass123')
-await page.click('#«r5»-button')
-```
-
-## Testing
-
-The changes are backward compatible - the agent will still use ID selectors as a last resort when semantic selectors are not available. The build completed successfully with no linter errors.
-
-## Files Modified
-
-1. `/src/orchestrator/orchestrator-agent.ts` - System prompt enhancements
-2. `/src/utils/page-info-utils.ts` - Selector priority reordering
-3. `/src/orchestrator/tools/take-screenshot.ts` - Vision analysis updates
-
-## Next Steps
-
-Test the changes with real scenarios to verify that:
-1. Generated commands use semantic selectors when available
-2. Tests remain stable and maintainable
-3. Fallback to ID selectors works when semantic options aren't available
-

package/plandocs/SESSION_SUMMARY_v0.0.33.md

@@ -1,151 +0,0 @@
-# Runner-Core v0.0.33 - Session Summary
-
-## Date: October 15, 2025
-
-## Major Accomplishments:
-
-### 1. ✅ **Coordinate Fallback System** (Phase 1 Complete)
-- Percentage-based coordinates (0-100%, 3 decimal precision)
-- Activates after 3 selector failures
-- 2 coordinate attempts before giving up
-- Resolution-independent positioning
-
-### 2. ✅ **Note to Future Self** (Tactical Memory)
-- Free-form notes persist across iterations AND steps
-- Enables strategic planning across agent decisions
-- Helps maintain context: "Tried X, will try Y next"
-
-### 3. ✅ **Visual Verification Tool** (NEW)
-- `verify_action_result` - Before/after screenshot comparison
-- Agent-callable (decides when to use)
-- JPEG 60% quality (85-90% smaller than PNG)
-- Multi-image LLM interface support
-
-### 4. ✅ **Critical Bug Fixes**
-- **Coordinate mode never activated**: Changed forced stuck from >= 3 to >= 5 failures
-- **Missing required fields**: Made parser flexible (accepts reasoning OR statusReasoning)
-- **Navigation timeouts**: Added 30s timeout guidance for page.goto()
-- **Strict mode violations**: Added scoping guidance (locator('#parent').getByText())
-
-### 5. ✅ **Prompt Optimizations**
-- **59% reduction**: 17,573 chars → 7,287 chars in system prompt
-- **Cache-optimized**: Static content first, dynamic last
-- **Cost savings**: ~40% overall with model tiering
-- **Focused on cognition**: Removed bloat, kept decision-making guidance
-
-### 6. ✅ **Model Optimization**
-- **gpt-5-mini**: Complex tasks (4 operations)
-  - Command generation
-  - Goal completion checks
-  - Repair suggestions
-  - Agent orchestration
-- **gpt-4o-mini**: Simple tasks (7 operations)
-  - Scenario breakdown
-  - Screenshot need assessment
-  - Repair confidence
-  - Test name generation
-  - Hashtag generation
-  - Script parsing
-  - Final script merging
-- **Est. 25-30% cost reduction**
-
-### 7. ✅ **Code Cleanup**
-- Removed V1 SmartTestRunnerCore (V2 is stable)
-- Removed backup files (.bak, .tmp)
-- Consolidated types into V2
-- Removed PeopleHR-specific examples from prompts
-
-### 8. ✅ **Enhanced Logging**
-- Prompt length metrics (chars + estimated tokens)
-- Full LLM response on parsing errors
-- Field presence diagnostics
-- Retry logging for 500 errors
-
-### 9. ✅ **Retry Logic**
-- Automatic retry for OpenAI 500 errors
-- Exponential backoff (1s, 2s, 4s)
-- Up to 3 attempts before failing
-
-### 10. ✅ **Headed Mode for Local Testing**
-- All browser instances use headed: false → headed: false for local dev
-- Visual debugging enabled
-
-## Files Modified:
-
-### Runner-Core:
-1. `src/orchestrator/orchestrator-agent.ts` - Main agent logic
-2. `src/orchestrator/types.ts` - NoteToFutureSelf, CoordinateAction
-3. `src/utils/coordinate-converter.ts` - NEW - Coordinate to Playwright conversion
-4. `src/orchestrator/tools/verify-action-result.ts` - NEW - Visual verification tool
-5. `src/llm-provider.ts` - Added LabeledImage, multi-image support
-6. `src/llm-facade.ts` - Model optimization
-7. `src/model-constants.ts` - Added DEFAULT_SIMPLER_MODEL
-8. `src/scenario-worker-class.ts` - Tool registration
-9. `src/orchestrator/index.ts` - Exports
-10. `src/orchestrator/tools/index.ts` - Tool exports
-
-### Scriptservice:
-1. `providers/scriptservice-llm-provider.ts` - Multi-image handling, retry logic
-2. `smart-test-runner-core-v2.ts` - Type definitions, V1 removal
-3. `smart-test-execution-handler.ts` - V1 removal
-4. `workers/test-based-explorer.ts` - V1 removal
-5. `script-generation-handlers.ts` - Headed mode
-6. `script-generation/script-generation-service.ts` - Headed mode
-7. `smart-test-execution-handler.ts` - Headed mode
-
-### Documentation:
-1. `WHATS_NEW_v0.0.33.md`
-2. `PHASE_1_COMPLETE.md`
-3. `PHASE_1_SUMMARY.md`
-4. `IMPLEMENTATION_STATUS.md`
-5. `VISUAL_AGENT_EVOLUTION_PLAN.md`
-6. `PROMPT_SANITY_CHECK.md`
-7. `PROMPT_OPTIMIZATION_ANALYSIS.md`
-8. `COORDINATE_MODE_DIAGNOSIS.md`
-9. `BEFORE_AFTER_VERIFICATION.md`
-10. `TROUBLESHOOTING_SESSION.md`
-
-## Live Test Status:
-
-**Job**: `71b88c60-52f5-4343-aef8-c44ebb07f3e9`
-**Status**: Running (check browser + logs)
-**Watch For**:
-- Step 5 (Employee Information) - Previously problematic
-- Coordinate mode activation
-- verify_action_result tool usage
-- Overall completion
-
-## Key Metrics:
-
-**Cost Optimization:**
-- Prompt size: 59% reduction
-- Model tiering: 7/11 tasks on cheaper model
-- JPEG compression: 85-90% smaller screenshots
-- **Total savings: ~40% cost reduction**
-
-## Next Steps After Test Completes:
-
-1. Check if Step 5 completes successfully
-2. Verify coordinate mode activated if needed
-3. Check if verify_action_result tool was used
-4. Analyze any remaining failures
-5. Iterate on prompts/logic based on results
-
-## Known Issues to Monitor:
-
-1. **Step 5 False Positive**: Clicking menu item vs navigating to page
-2. **Coordinate Loop**: Agent not knowing when coordinate clicks succeed
-3. **Vision verification usage**: Will agent call it proactively?
-
-## Success Criteria:
-
-✅ All 7 steps complete
-✅ Coordinate fallback used when selectors fail
-✅ Visual verification validates goal achievement
-✅ No infinite loops or stuck states
-✅ Generated script is accurate
-
----
-
-**Check your browser window and /tmp/scriptservice-test.log for live execution!**
-

package/plandocs/TROUBLESHOOTING_SESSION.md

@@ -1,72 +0,0 @@
-# Troubleshooting Session: All Modules Icon Click Failure
-
-## Objective:
-Understand why the orchestrator agent gets stuck on "Click on the all Modules menu item (top menu icon)" while manual Playwright MCP navigation succeeded.
-
-## What I Need to See:
-
-### 1. Full Agent Logs for the Failing Step
-Please provide the complete logs showing:
-- What iteration attempts were made (iteration 1, 2, 3...)
-- What selectors the agent tried each time
-- What errors it encountered
-- What the DOM snapshot showed
-- Whether it took screenshots
-- What notes it left to future self
-
-### 2. The DOM Context It Saw
-- Interactive elements list
-- ARIA tree snapshot
-- Whether the hamburger icon was visible in the list
-
-## What Worked (My Manual MCP Session):
-
-From earlier successful navigation:
-```
-✅ Step 1: Clicked hamburger menu
-   Selector: #sidebar-toggle > span > svg
-   
-✅ Step 2: Clicked "Core HR"  
-   Selector: getByText('Core HR')
-   
-✅ Step 3: Clicked "Employee Information"
-   Selector: getByText('Employee Information')
-```
-
-## Hypothesis of Why Agent Fails:
-
-### Possible Issue 1: Wrong Selector Strategy
-- Agent might be trying: `getByText('All Modules')` (strict mode violation)
-- Or: `#MenuToggle` (wrong ID)
-- Or: `#sidebar-toggle-menu` (doesn't exist)
-- Instead of: `#sidebar-toggle > span > svg` (actual selector)
-
-### Possible Issue 2: Missing Icon Detection
-- Hamburger icons are often SVG elements without accessible text
-- Agent might not recognize this pattern
-- Prompt doesn't explicitly guide on icon/SVG selector strategy
-
-### Possible Issue 3: DOM List Incomplete
-- Interactive elements might not include the SVG icon
-- If icon isn't in the list, agent won't know it exists
-- Need to check if `getEnhancedPageInfo` captures SVG icons
-
-### Possible Issue 4: Ambiguous Text
-- "All Modules" might appear in multiple places (menu button + modal title)
-- Agent tries `getByText('All Modules')` → strict mode violation
-- Should scope to parent: `locator('#sidebar-toggle').getByText('All Modules')`
-
-## Next Steps:
-
-1. **Get full logs** from your failing run
-2. **Compare** what agent saw vs what I saw  
-3. **Identify** the gap (prompt, DOM extraction, or selector logic)
-4. **Plan fixes**:
-   - Prompt improvements (icon/SVG guidance)
-   - DOM extraction improvements (ensure icons are captured)
-   - Selector strategy improvements (parent scoping for icons)
-   - Example-based learning (hamburger menu pattern)
-
-## Waiting For:
-Please paste the full logs from the failing step showing all iteration attempts and what the agent tried.
-

package/plandocs/VISION_DIAGNOSTICS_IMPROVEMENTS.md

@@ -1,336 +0,0 @@
-# Vision-Based Diagnostic Analysis Improvements
-
-## Overview
-Enhanced the test automation system to use screenshot-based vision analysis as a **diagnostic tool** to understand WHY failures occur and recommend better strategies based on visual reality vs DOM assumptions.
-
-**Vision diagnostics are now utilized in BOTH:**
-1. **Script Generation** (`ScenarioWorker`) - When generating commands from scenarios
-2. **Script Repair/Execution** (`ExecutionService`) - When repairing failing scripts with AI
-
-## Problem Statement (From Logs)
-1. **Hallucinated Verification Targets**: Goal completion was creating fake sub-goals like "verify message was sent" and looking for non-existent elements like "Message sent" text
-2. **No Screenshot Analysis**: Vision mode was never triggering because LLM assessment always said "NO"
-
-## Solutions Implemented
-
-### 1. Fixed Hallucinated Verification Targets ✅
-
-#### A. Goal Completion Check (`prompts.ts` lines 55-129)
-**Changes:**
-- System prompt: "Be EXTREMELY CONSERVATIVE - mark goals complete when PRIMARY action succeeds. DO NOT invent verification steps"
-- Added golden rule: "If goal is SIMPLE ACTION and action SUCCEEDED, mark COMPLETE immediately"
-- Explicit examples showing action goals complete after action succeeds (no verification needed)
-
-**Example:**
-```
-Before: "Click send button" → click succeeds → "INCOMPLETE - need to verify message sent"
-After:  "Click send button" → click succeeds → "COMPLETE ✅"
-```
-
-#### B. Command Generation Anti-Hallucination (`prompts.ts` lines 215-225)
-**New Section: "NEVER Hallucinate Verification Elements"**
-- ONLY verify elements that ACTUALLY EXIST in current DOM state
-- Don't invent success messages, confirmations, or sent indicators
-- Use alternative verification: state changes, network, page load
-- Stop trying to find elements after previous attempts failed
-
-#### C. Smart Hallucination Detection (`llm-facade.ts` lines 573-599)
-**Automatic Detection:**
-- Detects when LLM repeatedly tries to find non-existent elements (2+ attempts)
-- Shows "⚠️ HALLUCINATION ALERT" with guidance to stop and use alternatives
-- Analyzes command patterns (getByText, toBeVisible) + errors (not found, timeout)
-
-### 2. Enabled Screenshot-Based Diagnostic Analysis 📸
-
-#### A. Conservative Screenshot Trigger (`scenario-worker-class.ts` line 193)
-```typescript
-// ONLY on attempt 2 (3rd attempt, after exactly 2 failures)
-// This is the ONLY chance to use vision - must be absolutely necessary
-if (attempt === 2 && lastError && !usedVisionMode)
-```
-
-**Attempt Flow:**
-- Attempt 0: First try (DOM-based)
-- Attempt 1: Second try (DOM-based) 
-- **Attempt 2: Third try - Vision assessment (if truly needed)**
-- Attempt 3: Fourth try (final, DOM-based)
-
-#### B. Conservative Screenshot Assessment (`prompts.ts` lines 131-191)
-**System Prompt:** "Be LIBERAL in recommending screenshots - visual context provides diagnostic insights DOM cannot"
-
-**Diagnostic Value Framework:**
-1. Identify WHY attempts failed (DOM assumptions vs visual reality)
-2. Detect hallucinated elements (see if expected elements exist visually)
-3. Recommend better strategies (verification based on what's visible)
-4. Find visual blockers (overlays, modals, layout issues)
-5. Correct wrong assumptions (actual state vs expected state)
-
-**Hallucination Detection in Assessment:**
-- If "not found/timeout" errors + verification attempts → HIGH chance of hallucination
-- Screenshot reveals if elements actually exist
-
-#### C. Enhanced Vision Mode as Diagnostic Tool (`prompts.ts` lines 359-462)
-
-**System Prompt:** "Analyze screenshot to understand WHY previous attempts failed and recommend BEST next step based on visual reality vs DOM assumptions"
-
-**Three Primary Tasks:**
-1. **DEBUG WHY PREVIOUS ATTEMPTS FAILED** - Compare what you assumed vs what you SEE
-2. **IDENTIFY THE ROOT CAUSE** - Why did DOM-based approaches fail?
-3. **RECOMMEND SMARTER ALTERNATIVES** - What should we do instead?
-
-**Critical Questions Framework:**
-
-🔍 **Visual vs DOM Reality Check:**
-- What do you SEE vs what DOM suggested?
-- Are elements you tried to find VISIBLE on screen?
-- Are there visual indicators you missed?
-- What's ACTUALLY present vs ASSUMED?
-
-🚫 **Why Did Previous Attempts Fail?**
-- Looking for elements that DON'T EXIST? (hallucination)
-- Wrong selectors for elements that ARE visible?
-- Elements blocked/covered by overlays?
-- Elements in different state than expected?
-- Page in different state than assumed?
-
-💡 **What Should You Do Differently?**
-- If verification elements don't exist: Use state-based verification
-- If elements blocked: Remove blocker first
-- If wrong selector: Use visual clues for better selectors
-- If goal achieved: Confirm with simple wait/check
-
-**Verification Strategy Based on Visual Reality:**
-
-```
-IF you see success indicators in screenshot:
-  ✅ Use them: await expect(page.getByText('visible-text-here')).toBeVisible()
-
-IF you DON'T see success indicators but action appears complete:
-  ✅ Use state-based checks:
-     - Button disabled check
-     - Form cleared/reset check
-     - URL changed check
-     - Network response verification
-     - Load state verification
-
-IF previous attempts looked for non-existent elements:
-  ❌ STOP looking for them
-  ✅ Switch to alternative verification
-```
-
-**Comparison Analysis Template:**
-```
-"Based on screenshot analysis:
- - DOM suggested: [what you thought was there]
- - Visual reality: [what you actually see]
- - Why previous failed: [root cause analysis]
- - Better approach: [what to do instead]"
-```
-
-#### D. Enhanced Diagnostic Logging (`llm-facade.ts` lines 313-328)
-
-**Vision Mode Response Now Includes:**
-- `visualInsights`: What screenshot revealed that DOM couldn't tell you
-- `failureRootCause`: Why previous DOM-based attempts failed
-- `recommendedAlternative`: Better verification/interaction strategy
-- `reasoning`: Full diagnostic analysis
-
-**Console Logging:**
-```
-📸 Visual insights: [what was discovered]
-🔍 Root cause analysis: [why it failed]
-💡 Recommended alternative: [what to do instead]
-🧠 Vision-based reasoning: [full analysis]
-```
-
-## Code Modularization
-
-Vision diagnostics are properly modularized and reused across both flows:
-
-1. **`LLMFacade` Methods** (shared by both flows):
-   - `assessScreenshotNeed()` - Conservative assessment if screenshot would help
-   - `getVisionDiagnostics()` - Supervisor analyzes screenshot (gpt-4o)
-   - `generateCommandFromSupervisorInstructions()` - Worker generates command (gpt-4.1-mini)
-
-2. **Two-Step Supervisor Pattern** (used consistently):
-   ```typescript
-   // Step 1: Supervisor analyzes screenshot (expensive vision model)
-   const supervisorDiagnostics = await llmFacade.getVisionDiagnostics(...)
-   
-   // Step 2: Use insights for action
-   // - In generation: Generate command from instructions
-   // - In repair: Enhance failure context with visual insights
-   ```
-
-3. **No Code Duplication**:
-   - Vision prompts defined once in `prompts.ts`
-   - LLM calls handled by `LLMFacade`
-   - Both flows use same vision assessment logic
-
-## Behavioral Changes
-
-### Script Generation - Before:
-```
-1. Click send button → succeeds
-2. Goal check → "INCOMPLETE - verify message sent" (hallucinated)
-3. Try: await expect(page.getByText('Message sent')).toBeVisible()
-4. Fail: Element not found
-5. Try: await page.getByRole('status').waitFor()
-6. Fail: Timeout
-7. Repeat until max failures
-8. Screenshot assessment → "NO, not needed"
-9. Vision mode → never triggers
-```
-
-### Script Generation - After:
-```
-1. Click send button → succeeds
-2. Goal check → "COMPLETE ✅" (action succeeded, no verification needed)
-   OR if verification truly needed but wrong approach:
-3. After 2 failures → Screenshot assessment
-4. Assessment → "YES - Diagnostic value: Visual reveals if success indicators exist"
-5. Vision mode activates with screenshot
-6. Diagnostic analysis:
-   - "DOM suggested: Success message would appear"
-   - "Visual reality: No success message visible, button is now disabled"
-   - "Root cause: Hallucinated verification element that doesn't exist"
-   - "Better approach: Check button disabled state instead"
-7. Generates: await expect(page.locator('button[name="Send"]')).toBeDisabled()
-8. Success ✅
-```
-
-### Script Repair - Before:
-```
-1. Script step fails: await page.click('button[name="Submit"]')
-2. AI Repair attempt 1: Try different selector → Fails
-3. AI Repair attempt 2: Try with wait → Fails
-4. AI Repair attempt 3: Try another approach → Fails
-5. Give up - repair failed
-```
-
-### Script Repair - After:
-```
-1. Script step fails: await page.click('button[name="Submit"]')
-2. AI Repair attempt 1: Try different selector → Fails
-3. AI Repair attempt 2: Try with wait → Fails
-4. After 2 failures → Screenshot assessment
-5. Assessment → "YES - Visual analysis can reveal why repairs are failing"
-6. Vision supervisor analyzes screenshot:
-   - "Visual analysis: Button is disabled and grayed out"
-   - "Root cause: Trying to click a disabled button"
-   - "Recommended approach: Wait for button to become enabled first"
-7. AI Repair attempt 3 with vision insights: Insert wait step before click
-8. Success ✅ (vision-aided repair)
-```
-
-## Cost Considerations
-
-**Vision Mode (GPT-4o) Triggers in Both Flows:**
-
-1. **Script Generation** (`ScenarioWorker`):
-   - After 2 failed command generation attempts (3rd attempt)
-   - Only when LLM assessment says "YES"
-   - Prevents endless DOM-based retries on wrong assumptions
-
-2. **Script Repair** (`ExecutionService`):
-   - After 2 failed repair attempts (3rd/final repair attempt)
-   - Only when LLM assessment says "YES"
-   - Prevents wasted repair cycles on wrong strategy
-
-**What You Get (Both Flows):**
-- Root cause analysis of failures
-- Visual vs DOM reality comparison
-- Recommended alternative strategies
-- Smart fallback only when truly needed
-- Prevents wasted attempts on wrong approaches
-
-## Files Modified
-
-1. **`/runner-core/src/prompts.ts`**
-   - Enhanced goal completion check (lines 55-129)
-   - Enhanced screenshot assessment (lines 131-184)
-   - Enhanced vision mode prompt with diagnostics (lines 359-462)
-   - Added anti-hallucination sections
-
-2. **`/runner-core/src/llm-facade.ts`**
-   - Added hallucination detection (lines 573-599)
-   - Enhanced vision logging (lines 313-328)
-   - Vision methods: `assessScreenshotNeed`, `getVisionDiagnostics`, `generateCommandFromSupervisorInstructions`
-
-3. **`/runner-core/src/scenario-worker-class.ts`** (Script Generation)
-   - Vision fallback on attempt 2 (line 197)
-   - Two-step supervisor pattern for command generation (lines 197-241)
-
-4. **`/runner-core/src/execution-service.ts`** (Script Repair/Execution)
-   - Vision fallback on final repair attempt after 2 failures (lines 500-594)
-   - Vision-enhanced context for repair suggestions
-   - Same supervisor pattern for consistency
-
-## Testing Recommendations
-
-### Script Generation Tests
-
-1. **Test with messaging flow:**
-   - Verify "send message" steps complete without hallucinated verification
-   - Check if vision mode provides diagnostic insights when command generation fails
-
-2. **Test with form submission:**
-   - Ensure submit actions complete without looking for non-existent confirmations
-   - Verify alternative verification strategies are used
-
-### Script Repair Tests
-
-1. **Test with selector failures:**
-   - Run script with outdated selectors
-   - Verify vision diagnostics reveal actual element state
-   - Check that repairs use vision insights
-
-2. **Test with state-dependent failures:**
-   - Run script where timing/state causes failures
-   - Verify vision reveals actual page state (e.g., disabled buttons, loading states)
-   - Check that repairs address root cause
-
-### Monitor Both Flows
-
-1. **Vision mode activation:**
-   - Should trigger after 2 failures when truly needed (not for every failure)
-   - Check diagnostic logs for root cause analysis
-   - Verify recommended alternatives are sensible
-
-2. **Success patterns:**
-   - Look for "(vision-aided)" markers in success logs
-   - Track improvement in success rate after vision diagnostics added
-
-## Next Steps
-
-1. **Monitor logs for:**
-   - Reduced hallucination instances (generation flow)
-   - Vision mode diagnostic quality (both flows)
-   - Alternative verification strategies effectiveness (generation flow)
-   - Repair success rate improvement (repair flow)
-
-2. **Metrics to track:**
-   - Vision mode activation rate in generation vs repair
-   - Hallucination detection rate
-   - Test success rate improvement (generation)
-   - Repair success rate improvement (execution)
-   - Cost vs benefit of vision mode in each flow
-
-3. **Potential future enhancements:**
-   - Learn from vision diagnostics to improve DOM-only approaches
-   - Build a library of common visual patterns and solutions
-   - Optimize screenshot timing based on diagnostic value
-   - Consider vision diagnostics for other failure scenarios
-   - Share vision insights between generation and repair flows
-
-## Summary
-
-**✅ Vision-based diagnostics are now properly integrated into BOTH:**
-- **Script Generation** - Helps generate better commands when DOM info insufficient
-- **Script Repair** - Helps diagnose why repairs fail and suggest better fixes
-
-**✅ Properly modularized with no code duplication:**
-- Shared `LLMFacade` methods
-- Consistent two-step supervisor pattern
-- Single source of truth for prompts and logic
-