testchimp-runner-core 0.0.21 → 0.0.23

package/plandocs/SCRIPT_GENERATION_ARCHITECTURE.md

@@ -0,0 +1,364 @@
+# Script Generation Architecture & Work Plan
+
+## Overview
+AI-powered test script generation from natural language scenarios using LLM-guided Playwright automation with vision-based fallback diagnostics.
+
+## Architecture Flow
+
+```
+User Scenario (text file)
+    ↓
+1. Scenario Breakdown (LLM)
+    ↓
+2. Step-by-Step Execution
+    ↓
+3. Command Generation (LLM + DOM)
+    ↓
+4. Playwright Execution
+    ↓
+5. Goal Completion Check (LLM)
+    ↓
+6. Vision Fallback (if needed)
+    ↓
+7. Script Generation
+```
+
+## Components
+
+### 1. Scenario Breakdown
+**File:** `llm-facade.ts` → `breakdownScenario()`  
+**Prompt:** `PROMPTS.SCENARIO_BREAKDOWN`
+
+**Input:** Natural language scenario
+```
+- Go to https://app.com
+- Login with credentials: admin, pass123
+- Click on settings
+```
+
+**Output:** Structured steps
+```json
+{
+  "steps": [
+    "Go to https://app.com",
+    "Login with credentials: admin, pass123",
+    "Click on settings"
+  ]
+}
+```
+
+**Key Principles:**
+- ✅ Preserve ALL specific values (credentials, names, amounts, etc.)
+- ✅ Keep steps semantic (no technical selectors)
+- ✅ One clear action per step
+- ❌ Never replace values with variables/placeholders
+
+### 2. Step Execution Loop
+**File:** `scenario-worker-class.ts` → `processScenarioJob()`
+
+**For each step:**
+1. Initialize step tracking
+2. Execute sub-actions until goal complete
+3. Track failures and successes
+4. Generate final script
+
+**Counters:**
+- `subActionCount`: Number of different commands tried for this step
+- `totalFailedAttemptsForStep`: Total failures across all sub-actions
+- `attempt`: Retry count within current sub-action (0-3)
+
+### 3. Command Generation
+**File:** `llm-facade.ts` → `generatePlaywrightCommand()`  
+**Prompt:** `PROMPTS.PLAYWRIGHT_COMMAND`
+
+**Context Provided:**
+- Goal description
+- Current page state (DOM snapshot)
+- Previous commands in this step
+- Previous step history
+- Last error (if retry)
+
+**Key Principles:**
+1. **Extract specific values from goal** - Use exact credentials, names, amounts from goal description
+2. **Navigation handling** - Use `{ waitUntil: 'domcontentloaded', timeout: 10000 }` for redirects
+3. **Check current URL** - Don't retry navigation if already navigated (even if redirected)
+4. **Never hallucinate verification** - Only verify what goal explicitly asks for
+5. **Semantic action completion** - "Login" means fill + click, not just fill
+
+### 4. Goal Completion Assessment
+**File:** `llm-facade.ts` → `checkGoalCompletion()`  
+**Prompt:** `PROMPTS.GOAL_COMPLETION_CHECK`
+
+**Decision Matrix:**
+
+| Goal Type | Completion Criteria | Example |
+|-----------|-------------------|---------|
+| Simple action | Action succeeded | "Click button" → complete after click |
+| Semantic action | All implicit steps done | "Login" → complete after fill + click |
+| Multi-part action | All parts done | "Fill all fields" → complete after all fields |
+| Verification | Assertion passed | "Verify message" → complete after assertion |
+
+**Semantic Action Recognition:**
+- **"Login with credentials"** → Fill username, fill password, click login button
+- **"Send message"** → Type message, click send button
+- **"Submit form"** → Fill fields, click submit button
+- **"Register/Signup"** → Fill registration, click register button
+
+Mark INCOMPLETE until the final implicit action completes.
+
+### 5. Vision-Based Fallback Diagnostics
+**File:** `scenario-worker-class.ts` (lines 215-272)  
+**Prompts:** `SCREENSHOT_NEED_ASSESSMENT`, `VISION_DIAGNOSTIC_ANALYSIS`
+
+**Trigger Condition:**
+```typescript
+totalFailedAttemptsForStep >= 2 && !usedVisionMode && lastError
+```
+
+**When:** After 2+ total failures across all sub-actions
+
+**Two-Step Process:**
+
+**Step 1: Assess Screenshot Need** (gpt-4.1-mini)
+- Quick check: Would visual analysis help?
+- Conservative: Only recommend if DOM info insufficient
+- Returns: needsScreenshot (boolean) + reason
+
+**Step 2: Vision Diagnostics** (gpt-4o - only if assessment says yes)
+- Supervisor analyzes screenshot
+- Identifies: What's visible vs what was assumed
+- Diagnoses: Why previous attempts failed
+- Recommends: Better approach based on visual reality
+
+**Output:**
+- Visual analysis
+- Root cause of failures
+- Specific instructions for next attempt
+- Elements found/not found
+
+### 6. Script Generation
+**File:** `script-utils.ts` → `generateTestScript()`
+
+**Output Format:**
+```javascript
+/*
+This is a TestChimp Smart Test.
+Version: 1.0
+
+#login #coreHR #peopleHR
+*/
+
+import { test, expect } from '@playwright/test';
+test('testName', async ({ page, browser, context }) => {
+  // Step 1: Go to URL
+  await page.goto('https://...', { waitUntil: 'domcontentloaded', timeout: 10000 });
+  
+  // Step 2: Login with credentials: Willy, Willy@1234
+  await page.fill('username', 'Willy');
+  await page.fill('password', 'Willy@1234');
+  await page.click('button[name="Login"]');
+  
+  // Step 3: Click on All Modules [FAILED]
+  // Attempted: await page.getByText('All Modules').click();
+});
+```
+
+## Configuration & Timeouts
+
+**Default Timeout:** 5 seconds (fast feedback on wrong selectors)
+
+**Navigation Timeout:** 10 seconds explicit (handles redirects)
+```typescript
+await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 10000 })
+```
+
+**Why:**
+- 5s for element operations → fast failure on wrong selectors (not 10s wait per wrong selector)
+- 10s explicit for navigation → handles redirects properly
+- Best of both: fast iteration + reliable navigation
+
+## Key Improvements
+
+### 1. Value Preservation Throughout Flow
+
+**Problem:** Losing specific values (credentials, amounts, etc.)
+
+**Solution:** Preserve at every stage
+
+| Stage | Before | After |
+|-------|--------|-------|
+| Breakdown | "Login with user/pass" | "Login with credentials: Willy, Willy@1234" |
+| Goal | "Complete login" | "Login with credentials: Willy, Willy@1234" |
+| Command | `process.env.USERNAME` | `'Willy'` |
+
+### 2. Semantic Action Understanding
+
+**Problem:** Marking "Login" complete after just filling fields
+
+**Solution:** Recognize implicit final actions
+
+- "Login" → fill + **click login button**
+- "Send" → type + **click send button**
+- "Submit" → fill + **click submit button**
+
+### 3. Navigation & Redirect Handling
+
+**Problem:** Retrying original URL after successful redirect
+
+**Solution:** 
+- Check current URL after navigation errors
+- If URL changed from `about:blank` → navigation succeeded
+- Use `domcontentloaded` for redirects (more reliable than `load`)
+- Don't retry if already on a page
+
+### 4. Vision Diagnostics
+
+**Problem:** Vision never triggering (was checking per-sub-action attempt)
+
+**Solution:** Trigger on total failures across all sub-actions
+- Changed from `attempt === 2` → `totalFailedAttemptsForStep >= 2`
+- Now triggers after 2+ failures regardless of sub-action boundaries
+- Detailed logging shows when/why vision triggers or doesn't
+
+### 5. Enhanced Logging
+
+**Visibility:**
+- ✅ Console logs for Debug Console
+- ✅ outputChannel for Output panel
+- ✅ Timestamps on all logs
+- ✅ Version markers
+- ✅ Vision trigger decision logs
+
+**Format:**
+```
+[02:58:15.613] [ScenarioWorker] 🚀 RUNNER-CORE VERSION: v1.5.0-vision-preserve-values
+[02:58:15.614] [ScenarioWorker] Step 1 - Sub-action 1, Attempt 1: Go to URL
+[02:58:15.650] [ScenarioWorker]   🔍 Vision trigger check: subAction=1, attempt=0, totalFailed=0, usedVision=false
+[02:58:15.651] [ScenarioWorker]   📝 Using DOM-based approach (0 failures so far, need 2+)
+```
+
+## Retry & Failure Budget
+
+**Per Step Limits:**
+- `MAX_RETRIES_PER_STEP = 3` → 4 attempts per sub-action (0, 1, 2, 3)
+- `MAX_SUBACTIONS_PER_STEP = 5` → Max 5 different commands for one step
+- `MAX_FAILED_ATTEMPTS_PER_STEP = 12` → Hard limit on total failures
+
+**Early Termination:**
+- After 2 consecutive step failures → stop execution
+- Saves resources, prevents runaway costs
+
+## Error Context Enhancement
+
+**Navigation errors now include current URL:**
+```
+Error: Timeout 10000ms exceeded | Current URL: https://redirected-url.com
+```
+
+This helps LLM understand:
+- Navigation succeeded but redirected
+- Don't retry original URL
+- Proceed with current page
+
+## Workflow Example
+
+**Scenario:**
+```
+- Go to https://app.com/login
+- Login with credentials: admin, pass123
+- Click dashboard
+```
+
+**Execution:**
+
+**Step 1: Navigate**
+```
+Attempt 1: goto(url, {domcontentloaded, timeout: 10000}) → ✅ Success
+Goal check: COMPLETE (navigation is single-step action)
+```
+
+**Step 2: Login**
+```
+Sub-action 1, Attempt 1: fill(username, 'admin') → ✅ Success
+Goal check: INCOMPLETE (login needs username + password + click)
+  nextSubGoal: "Enter password and click login"
+
+Sub-action 2, Attempt 1: fill(password, 'pass123') → ✅ Success  
+Goal check: INCOMPLETE (still need to click login button)
+  nextSubGoal: "Click login button to submit credentials"
+
+Sub-action 3, Attempt 1: click(login button) → ✅ Success
+Goal check: COMPLETE (all parts of login done)
+```
+
+**Step 3: Click dashboard**
+```
+Sub-action 1, Attempt 1: click(dashboard) → ❌ Fail (not visible)
+  🔍 Vision check: totalFailed=1, need 2+
+  📝 Using DOM (1 failure, need 2+)
+
+Sub-action 1, Attempt 2: waitFor + click → ❌ Fail (still not visible)
+  🔍 Vision check: totalFailed=2, usedVision=false
+  🎯 VISION TRIGGER: 2 total failures - assessing...
+  💭 LLM: SCREENSHOT NEEDED ✅
+  📸 Taking screenshot...
+  👔 Supervisor analyzing...
+  🔨 Generating vision-aided command...
+
+Sub-action 1, Attempt 3: [vision-aided command] → ✅ Success
+Goal check: COMPLETE
+```
+
+## Testing Checklist
+
+- [ ] Specific values preserved (credentials, names, amounts)
+- [ ] Semantic actions complete fully (login includes button click)
+- [ ] Navigation redirects handled (no URL retry loops)
+- [ ] Vision triggers after 2+ failures
+- [ ] Vision logs show decision reasoning
+- [ ] Timeouts appropriate (5s default, 10s navigation)
+- [ ] Error context includes current URL
+- [ ] Failed steps don't show previous step commands
+- [ ] Version marker visible in logs
+
+## Version Tracking
+
+**Current Version:** `v1.5.0-vision-preserve-values`
+
+**Version log location:**
+- During initialization: `[ScenarioWorker] 🚀 RUNNER-CORE VERSION: v1.5.0-vision-preserve-values`
+- Increment for each significant change
+
+## Build & Deploy
+
+**Local Development:**
+```bash
+cd /Users/nuwansam/IdeaProjects/AwareRepo/local/vs-ext
+./build_local.sh
+```
+
+**What it does:**
+1. Builds runner-core
+2. Packs runner-core (0.0.22)
+3. Installs in vs-ext
+4. Builds vs-ext for staging
+
+**Verification:**
+```bash
+grep "v1.5.0-vision-preserve-values" node_modules/testchimp-runner-core/dist/scenario-worker-class.js
+```
+
+## Related Documentation
+
+- `VISION_DIAGNOSTICS_IMPROVEMENTS.md` - Vision system details
+- `prompts.ts` - All LLM prompts and guidance
+- `types.ts` - Type definitions
+
+## Future Enhancements
+
+1. **Learn from vision insights** - Build library of common patterns
+2. **Optimize vision timing** - Better cost/benefit analysis
+3. **Cross-flow learning** - Share insights between generation and repair
+4. **Smarter goal parsing** - Better semantic action recognition
+5. **Dynamic timeout adjustment** - Based on operation type
+

package/plandocs/SELECTOR_IMPROVEMENTS.md

@@ -0,0 +1,139 @@
+# Selector Preference Improvements
+
+## Summary
+Updated the orchestrator agent to prefer user-friendly, semantic Playwright selectors over auto-generated IDs, following Playwright's official best practices.
+
+## Problem
+The agent was generating commands like:
+```typescript
+await page.fill('#«r3»-form-item', 'alice@example.com')
+await page.fill('#«r4»-form-item', 'TestPass123')
+```
+
+These auto-generated IDs (especially with unicode characters like `«r3»`) are:
+- Not user-friendly or readable
+- Break when component instances change
+- Not maintainable
+- Not following Playwright best practices
+
+## Solution
+Implemented a comprehensive selector preference strategy across three key files:
+
+### 1. Orchestrator Agent Prompt (`orchestrator-agent.ts`)
+
+Added new section **"5b. SELECTOR PREFERENCE"** with explicit guidance:
+
+**Preferred selectors (in order):**
+1. `page.getByRole('role', {name: 'text'})` - Accessible, semantic, resilient
+2. `page.getByLabel('label text')` - Great for form inputs
+3. `page.getByPlaceholder('placeholder')` - Good for inputs without labels
+4. `page.getByText('visible text')` - Clear and readable
+5. `page.getByTestId('test-id')` - Stable if available
+
+**Avoid (last resort only):**
+- CSS selectors with auto-generated IDs: `#r3-form-item`, `#«r3»-form-item`
+- CSS selectors with unicode characters
+- Complex CSS paths
+
+**Examples provided:**
+```typescript
+// ❌ BAD
+await page.fill('#«r3»-form-item', 'alice@example.com')
+
+// ✅ GOOD
+await page.getByLabel('Email').fill('alice@example.com')
+await page.getByRole('textbox', {name: 'Email'}).fill('alice@example.com')
+await page.getByPlaceholder('Enter your email').fill('alice@example.com')
+```
+
+### 2. Page Info Utils (`page-info-utils.ts`)
+
+**Reordered selector generation priority:**
+
+Before:
+1. ID selector (e.g., `#«r3»-form-item`)
+2. data-testid
+3. getByRole
+
+After:
+1. getByLabel (for inputs with associated labels)
+2. getByRole with name (semantic and accessible)
+3. getByPlaceholder (for inputs with placeholders)
+4. getByText (visible text fallback)
+5. getByTestId (stable, explicit)
+6. ID selector - ONLY if stable (filters out auto-generated IDs with unicode or patterns like `rc_`, `:r[0-9]+:`, `__`)
+
+**Enhanced display:**
+- Shows best selector first, with up to 2 alternatives
+- Example: `getByLabel('Email') (or: getByRole('textbox', {name: 'Email'}), getByPlaceholder('Enter your email'))`
+
+### 3. Screenshot Tool (`take-screenshot.ts`)
+
+**Updated vision analysis prompts:**
+- System prompt now emphasizes: "ALWAYS prioritize semantic selectors (getByRole, getByLabel, getByText) over CSS selectors with auto-generated IDs"
+- Analysis task explicitly instructs: "Recommend SEMANTIC SELECTORS FIRST" and "AVOID auto-generated IDs with unicode"
+
+### 4. Updated EXPERIENCES Section
+
+Enhanced examples to capture semantic selector patterns:
+```
+✅ GOOD - App-specific patterns:
+- "Login form fields accessible via getByLabel: 'Email' and 'Password'"
+- "Submit buttons consistently use role=button with text matching action"
+- "Input fields have clear placeholders - prefer getByPlaceholder over IDs"
+
+❌ BAD:
+- Noting auto-generated IDs like #«r3»-form-item (these are unreliable)
+```
+
+## Benefits
+
+1. **More Maintainable**: Semantic selectors are resilient to UI changes
+2. **Self-Documenting**: Code reads like natural language
+3. **Accessibility**: Ensures UI elements are properly accessible
+4. **Best Practices**: Follows Playwright's official recommendations
+5. **User-Friendly**: Generated tests are more readable and easier to understand
+
+## Expected Output
+
+After these changes, the agent will generate:
+
+```typescript
+// Login form example
+await page.getByLabel('Email').fill('alice@example.com')
+await page.getByLabel('Password').fill('TestPass123')
+await page.getByRole('button', {name: 'Sign In'}).click()
+
+// Navigation example
+await page.getByRole('link', {name: 'Dashboard'}).click()
+
+// Form with placeholders
+await page.getByPlaceholder('Search...').fill('test query')
+```
+
+Instead of:
+
+```typescript
+// Old style with auto-generated IDs
+await page.fill('#«r3»-form-item', 'alice@example.com')
+await page.fill('#«r4»-form-item', 'TestPass123')
+await page.click('#«r5»-button')
+```
+
+## Testing
+
+The changes are backward compatible - the agent will still use ID selectors as a last resort when semantic selectors are not available. The build completed successfully with no linter errors.
+
+## Files Modified
+
+1. `/src/orchestrator/orchestrator-agent.ts` - System prompt enhancements
+2. `/src/utils/page-info-utils.ts` - Selector priority reordering
+3. `/src/orchestrator/tools/take-screenshot.ts` - Vision analysis updates
+
+## Next Steps
+
+Test the changes with real scenarios to verify that:
+1. Generated commands use semantic selectors when available
+2. Tests remain stable and maintainable
+3. Fallback to ID selectors works when semantic options aren't available
+

package/src/credit-usage-service.ts

@@ -22,6 +22,7 @@ export interface InsertCreditUsageResponse {
 export class CreditUsageService {
   private backendUrl: string;
   private authConfig: AuthConfig | null;
+  private logger?: (message: string, level?: 'log' | 'error' | 'warn') => void;
 
   constructor(authConfig?: AuthConfig, backendUrl?: string) {
     // Use provided backend URL or fall back to environment configuration
@@ -36,6 +37,23 @@ export class CreditUsageService {
     this.authConfig = authConfig || null;
   }
 
+  /**
+   * Set a logger callback for capturing execution logs
+   */
+  setLogger(logger: (message: string, level?: 'log' | 'error' | 'warn') => void): void {
+    this.logger = logger;
+  }
+
+  /**
+   * Log a message using the configured logger
+   */
+  private log(message: string, level: 'log' | 'error' | 'warn' = 'log'): void {
+    if (this.logger) {
+      this.logger(message, level);
+    }
+    // No console fallback - logs are routed to consumer
+  }
+
   /**
    * Update authentication configuration
    */
@@ -82,7 +100,7 @@ export class CreditUsageService {
 
       return response.data;
     } catch (error: any) {
-      console.error('Credit usage report failed:', error);
+      this.log(`Credit usage report failed: ${error}`, 'error');
       throw new Error(`Credit usage report failed: ${error.message}`);
     }
   }
@@ -93,9 +111,9 @@ export class CreditUsageService {
   async reportScriptGenerationCredit(jobId?: string): Promise<void> {
     try {
       await this.reportCreditUsage(1, CreditUsageReason.SCRIPT_GENERATE, jobId);
-      console.log(`Credit usage reported for script generation${jobId ? ` (job: ${jobId})` : ''}`);
+      this.log(`Credit usage reported for script generation${jobId ? ` (job: ${jobId})` : ''}`);
     } catch (error) {
-      console.error('Failed to report script generation credit usage:', error);
+      this.log(`Failed to report script generation credit usage: ${error}`, 'error');
       // Don't throw - credit reporting should not break the main flow
     }
   }
@@ -106,9 +124,9 @@ export class CreditUsageService {
   async reportAIRepairCredit(jobId?: string): Promise<void> {
     try {
       await this.reportCreditUsage(1, CreditUsageReason.TEST_REPAIR, jobId);
-      console.log(`Credit usage reported for AI repair${jobId ? ` (job: ${jobId})` : ''}`);
+      this.log(`Credit usage reported for AI repair${jobId ? ` (job: ${jobId})` : ''}`);
     } catch (error) {
-      console.error('Failed to report AI repair credit usage:', error);
+      this.log(`Failed to report AI repair credit usage: ${error}`, 'error');
       // Don't throw - credit reporting should not break the main flow
     }
   }