testchimp-runner-core 0.0.21 → 0.0.23

package/VISION_DIAGNOSTICS_IMPROVEMENTS.md

@@ -0,0 +1,336 @@
+# Vision-Based Diagnostic Analysis Improvements
+
+## Overview
+Enhanced the test automation system to use screenshot-based vision analysis as a **diagnostic tool** to understand WHY failures occur and recommend better strategies based on visual reality vs DOM assumptions.
+
+**Vision diagnostics are now utilized in BOTH:**
+1. **Script Generation** (`ScenarioWorker`) - When generating commands from scenarios
+2. **Script Repair/Execution** (`ExecutionService`) - When repairing failing scripts with AI
+
+## Problem Statement (From Logs)
+1. **Hallucinated Verification Targets**: Goal completion was creating fake sub-goals like "verify message was sent" and looking for non-existent elements like "Message sent" text
+2. **No Screenshot Analysis**: Vision mode was never triggering because LLM assessment always said "NO"
+
+## Solutions Implemented
+
+### 1. Fixed Hallucinated Verification Targets ✅
+
+#### A. Goal Completion Check (`prompts.ts` lines 55-129)
+**Changes:**
+- System prompt: "Be EXTREMELY CONSERVATIVE - mark goals complete when PRIMARY action succeeds. DO NOT invent verification steps"
+- Added golden rule: "If goal is SIMPLE ACTION and action SUCCEEDED, mark COMPLETE immediately"
+- Explicit examples showing action goals complete after action succeeds (no verification needed)
+
+**Example:**
+```
+Before: "Click send button" → click succeeds → "INCOMPLETE - need to verify message sent"
+After:  "Click send button" → click succeeds → "COMPLETE ✅"
+```
+
+#### B. Command Generation Anti-Hallucination (`prompts.ts` lines 215-225)
+**New Section: "NEVER Hallucinate Verification Elements"**
+- ONLY verify elements that ACTUALLY EXIST in current DOM state
+- Don't invent success messages, confirmations, or sent indicators
+- Use alternative verification: state changes, network, page load
+- Stop trying to find elements after previous attempts failed
+
+#### C. Smart Hallucination Detection (`llm-facade.ts` lines 573-599)
+**Automatic Detection:**
+- Detects when LLM repeatedly tries to find non-existent elements (2+ attempts)
+- Shows "⚠️ HALLUCINATION ALERT" with guidance to stop and use alternatives
+- Analyzes command patterns (getByText, toBeVisible) + errors (not found, timeout)
+
+### 2. Enabled Screenshot-Based Diagnostic Analysis 📸
+
+#### A. Conservative Screenshot Trigger (`scenario-worker-class.ts` line 193)
+```typescript
+// ONLY on attempt 2 (3rd attempt, after exactly 2 failures)
+// This is the ONLY chance to use vision - must be absolutely necessary
+if (attempt === 2 && lastError && !usedVisionMode)
+```
+
+**Attempt Flow:**
+- Attempt 0: First try (DOM-based)
+- Attempt 1: Second try (DOM-based) 
+- **Attempt 2: Third try - Vision assessment (if truly needed)**
+- Attempt 3: Fourth try (final, DOM-based)
+
+#### B. Conservative Screenshot Assessment (`prompts.ts` lines 131-191)
+**System Prompt:** "Be LIBERAL in recommending screenshots - visual context provides diagnostic insights DOM cannot"
+
+**Diagnostic Value Framework:**
+1. Identify WHY attempts failed (DOM assumptions vs visual reality)
+2. Detect hallucinated elements (see if expected elements exist visually)
+3. Recommend better strategies (verification based on what's visible)
+4. Find visual blockers (overlays, modals, layout issues)
+5. Correct wrong assumptions (actual state vs expected state)
+
+**Hallucination Detection in Assessment:**
+- If "not found/timeout" errors + verification attempts → HIGH chance of hallucination
+- Screenshot reveals if elements actually exist
+
+#### C. Enhanced Vision Mode as Diagnostic Tool (`prompts.ts` lines 359-462)
+
+**System Prompt:** "Analyze screenshot to understand WHY previous attempts failed and recommend BEST next step based on visual reality vs DOM assumptions"
+
+**Three Primary Tasks:**
+1. **DEBUG WHY PREVIOUS ATTEMPTS FAILED** - Compare what you assumed vs what you SEE
+2. **IDENTIFY THE ROOT CAUSE** - Why did DOM-based approaches fail?
+3. **RECOMMEND SMARTER ALTERNATIVES** - What should we do instead?
+
+**Critical Questions Framework:**
+
+🔍 **Visual vs DOM Reality Check:**
+- What do you SEE vs what DOM suggested?
+- Are elements you tried to find VISIBLE on screen?
+- Are there visual indicators you missed?
+- What's ACTUALLY present vs ASSUMED?
+
+🚫 **Why Did Previous Attempts Fail?**
+- Looking for elements that DON'T EXIST? (hallucination)
+- Wrong selectors for elements that ARE visible?
+- Elements blocked/covered by overlays?
+- Elements in different state than expected?
+- Page in different state than assumed?
+
+💡 **What Should You Do Differently?**
+- If verification elements don't exist: Use state-based verification
+- If elements blocked: Remove blocker first
+- If wrong selector: Use visual clues for better selectors
+- If goal achieved: Confirm with simple wait/check
+
+**Verification Strategy Based on Visual Reality:**
+
+```
+IF you see success indicators in screenshot:
+  ✅ Use them: await expect(page.getByText('visible-text-here')).toBeVisible()
+
+IF you DON'T see success indicators but action appears complete:
+  ✅ Use state-based checks:
+     - Button disabled check
+     - Form cleared/reset check
+     - URL changed check
+     - Network response verification
+     - Load state verification
+
+IF previous attempts looked for non-existent elements:
+  ❌ STOP looking for them
+  ✅ Switch to alternative verification
+```
+
+**Comparison Analysis Template:**
+```
+"Based on screenshot analysis:
+ - DOM suggested: [what you thought was there]
+ - Visual reality: [what you actually see]
+ - Why previous failed: [root cause analysis]
+ - Better approach: [what to do instead]"
+```
+
+#### D. Enhanced Diagnostic Logging (`llm-facade.ts` lines 313-328)
+
+**Vision Mode Response Now Includes:**
+- `visualInsights`: What screenshot revealed that DOM couldn't tell you
+- `failureRootCause`: Why previous DOM-based attempts failed
+- `recommendedAlternative`: Better verification/interaction strategy
+- `reasoning`: Full diagnostic analysis
+
+**Console Logging:**
+```
+📸 Visual insights: [what was discovered]
+🔍 Root cause analysis: [why it failed]
+💡 Recommended alternative: [what to do instead]
+🧠 Vision-based reasoning: [full analysis]
+```
+
+## Code Modularization
+
+Vision diagnostics are properly modularized and reused across both flows:
+
+1. **`LLMFacade` Methods** (shared by both flows):
+   - `assessScreenshotNeed()` - Conservative assessment if screenshot would help
+   - `getVisionDiagnostics()` - Supervisor analyzes screenshot (gpt-4o)
+   - `generateCommandFromSupervisorInstructions()` - Worker generates command (gpt-4.1-mini)
+
+2. **Two-Step Supervisor Pattern** (used consistently):
+   ```typescript
+   // Step 1: Supervisor analyzes screenshot (expensive vision model)
+   const supervisorDiagnostics = await llmFacade.getVisionDiagnostics(...)
+   
+   // Step 2: Use insights for action
+   // - In generation: Generate command from instructions
+   // - In repair: Enhance failure context with visual insights
+   ```
+
+3. **No Code Duplication**:
+   - Vision prompts defined once in `prompts.ts`
+   - LLM calls handled by `LLMFacade`
+   - Both flows use same vision assessment logic
+
+## Behavioral Changes
+
+### Script Generation - Before:
+```
+1. Click send button → succeeds
+2. Goal check → "INCOMPLETE - verify message sent" (hallucinated)
+3. Try: await expect(page.getByText('Message sent')).toBeVisible()
+4. Fail: Element not found
+5. Try: await page.getByRole('status').waitFor()
+6. Fail: Timeout
+7. Repeat until max failures
+8. Screenshot assessment → "NO, not needed"
+9. Vision mode → never triggers
+```
+
+### Script Generation - After:
+```
+1. Click send button → succeeds
+2. Goal check → "COMPLETE ✅" (action succeeded, no verification needed)
+   OR if verification truly needed but wrong approach:
+3. After 2 failures → Screenshot assessment
+4. Assessment → "YES - Diagnostic value: Visual reveals if success indicators exist"
+5. Vision mode activates with screenshot
+6. Diagnostic analysis:
+   - "DOM suggested: Success message would appear"
+   - "Visual reality: No success message visible, button is now disabled"
+   - "Root cause: Hallucinated verification element that doesn't exist"
+   - "Better approach: Check button disabled state instead"
+7. Generates: await expect(page.locator('button[name="Send"]')).toBeDisabled()
+8. Success ✅
+```
+
+### Script Repair - Before:
+```
+1. Script step fails: await page.click('button[name="Submit"]')
+2. AI Repair attempt 1: Try different selector → Fails
+3. AI Repair attempt 2: Try with wait → Fails
+4. AI Repair attempt 3: Try another approach → Fails
+5. Give up - repair failed
+```
+
+### Script Repair - After:
+```
+1. Script step fails: await page.click('button[name="Submit"]')
+2. AI Repair attempt 1: Try different selector → Fails
+3. AI Repair attempt 2: Try with wait → Fails
+4. After 2 failures → Screenshot assessment
+5. Assessment → "YES - Visual analysis can reveal why repairs are failing"
+6. Vision supervisor analyzes screenshot:
+   - "Visual analysis: Button is disabled and grayed out"
+   - "Root cause: Trying to click a disabled button"
+   - "Recommended approach: Wait for button to become enabled first"
+7. AI Repair attempt 3 with vision insights: Insert wait step before click
+8. Success ✅ (vision-aided repair)
+```
+
+## Cost Considerations
+
+**Vision Mode (GPT-4o) Triggers in Both Flows:**
+
+1. **Script Generation** (`ScenarioWorker`):
+   - After 2 failed command generation attempts (3rd attempt)
+   - Only when LLM assessment says "YES"
+   - Prevents endless DOM-based retries on wrong assumptions
+
+2. **Script Repair** (`ExecutionService`):
+   - After 2 failed repair attempts (3rd/final repair attempt)
+   - Only when LLM assessment says "YES"
+   - Prevents wasted repair cycles on wrong strategy
+
+**What You Get (Both Flows):**
+- Root cause analysis of failures
+- Visual vs DOM reality comparison
+- Recommended alternative strategies
+- Smart fallback only when truly needed
+- Prevents wasted attempts on wrong approaches
+
+## Files Modified
+
+1. **`/runner-core/src/prompts.ts`**
+   - Enhanced goal completion check (lines 55-129)
+   - Enhanced screenshot assessment (lines 131-184)
+   - Enhanced vision mode prompt with diagnostics (lines 359-462)
+   - Added anti-hallucination sections
+
+2. **`/runner-core/src/llm-facade.ts`**
+   - Added hallucination detection (lines 573-599)
+   - Enhanced vision logging (lines 313-328)
+   - Vision methods: `assessScreenshotNeed`, `getVisionDiagnostics`, `generateCommandFromSupervisorInstructions`
+
+3. **`/runner-core/src/scenario-worker-class.ts`** (Script Generation)
+   - Vision fallback on attempt 2 (line 197)
+   - Two-step supervisor pattern for command generation (lines 197-241)
+
+4. **`/runner-core/src/execution-service.ts`** (Script Repair/Execution)
+   - Vision fallback on final repair attempt after 2 failures (lines 500-594)
+   - Vision-enhanced context for repair suggestions
+   - Same supervisor pattern for consistency
+
+## Testing Recommendations
+
+### Script Generation Tests
+
+1. **Test with messaging flow:**
+   - Verify "send message" steps complete without hallucinated verification
+   - Check if vision mode provides diagnostic insights when command generation fails
+
+2. **Test with form submission:**
+   - Ensure submit actions complete without looking for non-existent confirmations
+   - Verify alternative verification strategies are used
+
+### Script Repair Tests
+
+1. **Test with selector failures:**
+   - Run script with outdated selectors
+   - Verify vision diagnostics reveal actual element state
+   - Check that repairs use vision insights
+
+2. **Test with state-dependent failures:**
+   - Run script where timing/state causes failures
+   - Verify vision reveals actual page state (e.g., disabled buttons, loading states)
+   - Check that repairs address root cause
+
+### Monitor Both Flows
+
+1. **Vision mode activation:**
+   - Should trigger after 2 failures when truly needed (not for every failure)
+   - Check diagnostic logs for root cause analysis
+   - Verify recommended alternatives are sensible
+
+2. **Success patterns:**
+   - Look for "(vision-aided)" markers in success logs
+   - Track improvement in success rate after vision diagnostics added
+
+## Next Steps
+
+1. **Monitor logs for:**
+   - Reduced hallucination instances (generation flow)
+   - Vision mode diagnostic quality (both flows)
+   - Alternative verification strategies effectiveness (generation flow)
+   - Repair success rate improvement (repair flow)
+
+2. **Metrics to track:**
+   - Vision mode activation rate in generation vs repair
+   - Hallucination detection rate
+   - Test success rate improvement (generation)
+   - Repair success rate improvement (execution)
+   - Cost vs benefit of vision mode in each flow
+
+3. **Potential future enhancements:**
+   - Learn from vision diagnostics to improve DOM-only approaches
+   - Build a library of common visual patterns and solutions
+   - Optimize screenshot timing based on diagnostic value
+   - Consider vision diagnostics for other failure scenarios
+   - Share vision insights between generation and repair flows
+
+## Summary
+
+**✅ Vision-based diagnostics are now properly integrated into BOTH:**
+- **Script Generation** - Helps generate better commands when DOM info insufficient
+- **Script Repair** - Helps diagnose why repairs fail and suggest better fixes
+
+**✅ Properly modularized with no code duplication:**
+- Shared `LLMFacade` methods
+- Consistent two-step supervisor pattern
+- Single source of truth for prompts and logic
+

package/dist/credit-usage-service.d.ts

@@ -15,7 +15,16 @@ export interface InsertCreditUsageResponse {
 export declare class CreditUsageService {
     private backendUrl;
     private authConfig;
+    private logger?;
     constructor(authConfig?: AuthConfig, backendUrl?: string);
+    /**
+     * Set a logger callback for capturing execution logs
+     */
+    setLogger(logger: (message: string, level?: 'log' | 'error' | 'warn') => void): void;
+    /**
+     * Log a message using the configured logger
+     */
+    private log;
     /**
      * Update authentication configuration
      */

package/dist/credit-usage-service.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"credit-usage-service.d.ts","sourceRoot":"","sources":["../src/credit-usage-service.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,UAAU,EAAkB,MAAM,eAAe,CAAC;AAG3D,oBAAY,iBAAiB;IAC3B,2BAA2B,IAAI;IAC/B,aAAa,IAAI;IACjB,WAAW,IAAI;IACf,eAAe,IAAI;CACpB;AAED,MAAM,WAAW,wBAAwB;IACvC,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,YAAY,CAAC,EAAE,iBAAiB,CAAC;IACjC,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,yBAAyB;CAEzC;AAED,qBAAa,kBAAkB;IAC7B,OAAO,CAAC,UAAU,CAAS;IAC3B,OAAO,CAAC,UAAU,CAAoB;gBAE1B,UAAU,CAAC,EAAE,UAAU,EAAE,UAAU,CAAC,EAAE,MAAM;IAaxD;;OAEG;IACH,aAAa,CAAC,UAAU,EAAE,UAAU,GAAG,IAAI;IAI3C;;OAEG;IACH,aAAa,IAAI,UAAU,GAAG,IAAI;IAIlC;;OAEG;IACG,iBAAiB,CACrB,OAAO,EAAE,MAAM,YAAI,EACnB,WAAW,EAAE,iBAAiB,EAC9B,KAAK,CAAC,EAAE,MAAM,GACb,OAAO,CAAC,yBAAyB,CAAC;IA8BrC;;OAEG;IACG,4BAA4B,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAUjE;;OAEG;IACG,oBAAoB,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;CAS1D"}
+{"version":3,"file":"credit-usage-service.d.ts","sourceRoot":"","sources":["../src/credit-usage-service.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,UAAU,EAAkB,MAAM,eAAe,CAAC;AAG3D,oBAAY,iBAAiB;IAC3B,2BAA2B,IAAI;IAC/B,aAAa,IAAI;IACjB,WAAW,IAAI;IACf,eAAe,IAAI;CACpB;AAED,MAAM,WAAW,wBAAwB;IACvC,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,YAAY,CAAC,EAAE,iBAAiB,CAAC;IACjC,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,yBAAyB;CAEzC;AAED,qBAAa,kBAAkB;IAC7B,OAAO,CAAC,UAAU,CAAS;IAC3B,OAAO,CAAC,UAAU,CAAoB;IACtC,OAAO,CAAC,MAAM,CAAC,CAA8D;gBAEjE,UAAU,CAAC,EAAE,UAAU,EAAE,UAAU,CAAC,EAAE,MAAM;IAaxD;;OAEG;IACH,SAAS,CAAC,MAAM,EAAE,CAAC,OAAO,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,KAAK,GAAG,OAAO,GAAG,MAAM,KAAK,IAAI,GAAG,IAAI;IAIpF;;OAEG;IACH,OAAO,CAAC,GAAG;IAOX;;OAEG;IACH,aAAa,CAAC,UAAU,EAAE,UAAU,GAAG,IAAI;IAI3C;;OAEG;IACH,aAAa,IAAI,UAAU,GAAG,IAAI;IAIlC;;OAEG;IACG,iBAAiB,CACrB,OAAO,EAAE,MAAM,YAAI,EACnB,WAAW,EAAE,iBAAiB,EAC9B,KAAK,CAAC,EAAE,MAAM,GACb,OAAO,CAAC,yBAAyB,CAAC;IA8BrC;;OAEG;IACG,4BAA4B,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAUjE;;OAEG;IACG,oBAAoB,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;CAS1D"}

package/dist/credit-usage-service.js

@@ -27,6 +27,21 @@ class CreditUsageService {
         }
         this.authConfig = authConfig || null;
     }
+    /**
+     * Set a logger callback for capturing execution logs
+     */
+    setLogger(logger) {
+        this.logger = logger;
+    }
+    /**
+     * Log a message using the configured logger
+     */
+    log(message, level = 'log') {
+        if (this.logger) {
+            this.logger(message, level);
+        }
+        // No console fallback - logs are routed to consumer
+    }
     /**
      * Update authentication configuration
      */
@@ -64,7 +79,7 @@ class CreditUsageService {
             return response.data;
         }
         catch (error) {
-            console.error('Credit usage report failed:', error);
+            this.log(`Credit usage report failed: ${error}`, 'error');
             throw new Error(`Credit usage report failed: ${error.message}`);
         }
     }
@@ -74,10 +89,10 @@ class CreditUsageService {
     async reportScriptGenerationCredit(jobId) {
         try {
             await this.reportCreditUsage(1, CreditUsageReason.SCRIPT_GENERATE, jobId);
-            console.log(`Credit usage reported for script generation${jobId ? ` (job: ${jobId})` : ''}`);
+            this.log(`Credit usage reported for script generation${jobId ? ` (job: ${jobId})` : ''}`);
         }
         catch (error) {
-            console.error('Failed to report script generation credit usage:', error);
+            this.log(`Failed to report script generation credit usage: ${error}`, 'error');
             // Don't throw - credit reporting should not break the main flow
         }
     }
@@ -87,10 +102,10 @@ class CreditUsageService {
     async reportAIRepairCredit(jobId) {
         try {
             await this.reportCreditUsage(1, CreditUsageReason.TEST_REPAIR, jobId);
-            console.log(`Credit usage reported for AI repair${jobId ? ` (job: ${jobId})` : ''}`);
+            this.log(`Credit usage reported for AI repair${jobId ? ` (job: ${jobId})` : ''}`);
         }
         catch (error) {
-            console.error('Failed to report AI repair credit usage:', error);
+            this.log(`Failed to report AI repair credit usage: ${error}`, 'error');
             // Don't throw - credit reporting should not break the main flow
         }
     }

package/dist/credit-usage-service.js.map

@@ -1 +1 @@
-{"version":3,"file":"credit-usage-service.js","sourceRoot":"","sources":["../src/credit-usage-service.ts"],"names":[],"mappings":";;;;;;AAAA,kDAA0B;AAC1B,+CAA2D;AAC3D,6CAA6C;AAE7C,IAAY,iBAKX;AALD,WAAY,iBAAiB;IAC3B,uGAA+B,CAAA;IAC/B,2EAAiB,CAAA;IACjB,uEAAe,CAAA;IACf,+EAAmB,CAAA;AACrB,CAAC,EALW,iBAAiB,iCAAjB,iBAAiB,QAK5B;AAYD,MAAa,kBAAkB;IAI7B,YAAY,UAAuB,EAAE,UAAmB;QACtD,qEAAqE;QACrE,IAAI,UAAU,EAAE,CAAC;YACf,IAAI,CAAC,UAAU,GAAG,UAAU,CAAC;QAC/B,CAAC;aAAM,CAAC;YACN,oEAAoE;YACpE,MAAM,SAAS,GAAG,IAAA,0BAAa,GAAE,CAAC;YAClC,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC,qBAAqB,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,UAAU,GAAG,UAAU,IAAI,IAAI,CAAC;IACvC,CAAC;IAED;;OAEG;IACH,aAAa,CAAC,UAAsB;QAClC,IAAI,CAAC,UAAU,GAAG,UAAU,CAAC;IAC/B,CAAC;IAED;;OAEG;IACH,aAAa;QACX,OAAO,IAAI,CAAC,UAAU,CAAC;IACzB,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,iBAAiB,CACrB,UAAkB,CAAC,EACnB,WAA8B,EAC9B,KAAc;QAEd,IAAI,CAAC,IAAI,CAAC,UAAU,EAAE,CAAC;YACrB,MAAM,IAAI,KAAK,CAAC,uEAAuE,CAAC,CAAC;QAC3F,CAAC;QAED,IAAI,CAAC;YACH,MAAM,WAAW,GAAG,IAAA,4BAAc,EAAC,IAAI,CAAC,UAAU,CAAC,CAAC;YACpD,MAAM,GAAG,GAAG,GAAG,IAAI,CAAC,UAAU,iCAAiC,CAAC;YAEhE,MAAM,OAAO,GAA6B;gBACxC,OAAO;gBACP,YAAY,EAAE,WAAW;gBACzB,MAAM,EAAE,KAAK;aACd,CAAC;YAEF,MAAM,QAAQ,GAAG,MAAM,eAAK,CAAC,IAAI,CAAC,GAAG,EAAE,OAAO,EAAE;gBAC9C,OAAO,EAAE;oBACP,GAAG,WAAW;oBACd,cAAc,EAAE,kBAAkB;iBACnC;gBACD,OAAO,EAAE,KAAK,CAAC,2CAA2C;aAC3D,CAAC,CAAC;YAEH,OAAO,QAAQ,CAAC,IAAI,CAAC;QACvB,CAAC;QAAC,OAAO,KAAU,EAAE,CAAC;YACpB,OAAO,CAAC,KAAK,CAAC,6BAA6B,EAAE,KAAK,CAAC,CAAC;YACpD,MAAM,IAAI,KAAK,CAAC,+BAA+B,KAAK,CAAC,OAAO,EAAE,CAAC,CAAC;QAClE,CAAC;IACH,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,4BAA4B,CAAC,KAAc;QAC/C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,iBAAiB,CAAC,CAAC,EAAE,iBAAiB,CAAC,eAAe,EAAE,KAAK,CAAC,CAAC;YAC1E,OAAO,CAAC,GAAG,CAAC,8CAA8C,KAAK,CAAC,CAAC,CAAC,UAAU,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QAC/F,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,KAAK,CAAC,kDAAkD,EAAE,KAAK,CAAC,CAAC;YACzE,gEAAgE;QAClE,CAAC;IACH,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,oBAAoB,CAAC,KAAc;QACvC,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,iBAAiB,CAAC,CAAC,EAAE,iBAAiB,CAAC,WAAW,EAAE,KAAK,CAAC,CAAC;YACtE,OAAO,CAAC,GAAG,CAAC,sCAAsC,KAAK,CAAC,CAAC,CAAC,UAAU,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QACvF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,KAAK,CAAC,0CAA0C,EAAE,KAAK,CAAC,CAAC;YACjE,gEAAgE;QAClE,CAAC;IACH,CAAC;CACF;AA7FD,gDA6FC"}
+{"version":3,"file":"credit-usage-service.js","sourceRoot":"","sources":["../src/credit-usage-service.ts"],"names":[],"mappings":";;;;;;AAAA,kDAA0B;AAC1B,+CAA2D;AAC3D,6CAA6C;AAE7C,IAAY,iBAKX;AALD,WAAY,iBAAiB;IAC3B,uGAA+B,CAAA;IAC/B,2EAAiB,CAAA;IACjB,uEAAe,CAAA;IACf,+EAAmB,CAAA;AACrB,CAAC,EALW,iBAAiB,iCAAjB,iBAAiB,QAK5B;AAYD,MAAa,kBAAkB;IAK7B,YAAY,UAAuB,EAAE,UAAmB;QACtD,qEAAqE;QACrE,IAAI,UAAU,EAAE,CAAC;YACf,IAAI,CAAC,UAAU,GAAG,UAAU,CAAC;QAC/B,CAAC;aAAM,CAAC;YACN,oEAAoE;YACpE,MAAM,SAAS,GAAG,IAAA,0BAAa,GAAE,CAAC;YAClC,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC,qBAAqB,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,UAAU,GAAG,UAAU,IAAI,IAAI,CAAC;IACvC,CAAC;IAED;;OAEG;IACH,SAAS,CAAC,MAAmE;QAC3E,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;IACvB,CAAC;IAED;;OAEG;IACK,GAAG,CAAC,OAAe,EAAE,QAAkC,KAAK;QAClE,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YAChB,IAAI,CAAC,MAAM,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC;QAC9B,CAAC;QACD,oDAAoD;IACtD,CAAC;IAED;;OAEG;IACH,aAAa,CAAC,UAAsB;QAClC,IAAI,CAAC,UAAU,GAAG,UAAU,CAAC;IAC/B,CAAC;IAED;;OAEG;IACH,aAAa;QACX,OAAO,IAAI,CAAC,UAAU,CAAC;IACzB,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,iBAAiB,CACrB,UAAkB,CAAC,EACnB,WAA8B,EAC9B,KAAc;QAEd,IAAI,CAAC,IAAI,CAAC,UAAU,EAAE,CAAC;YACrB,MAAM,IAAI,KAAK,CAAC,uEAAuE,CAAC,CAAC;QAC3F,CAAC;QAED,IAAI,CAAC;YACH,MAAM,WAAW,GAAG,IAAA,4BAAc,EAAC,IAAI,CAAC,UAAU,CAAC,CAAC;YACpD,MAAM,GAAG,GAAG,GAAG,IAAI,CAAC,UAAU,iCAAiC,CAAC;YAEhE,MAAM,OAAO,GAA6B;gBACxC,OAAO;gBACP,YAAY,EAAE,WAAW;gBACzB,MAAM,EAAE,KAAK;aACd,CAAC;YAEF,MAAM,QAAQ,GAAG,MAAM,eAAK,CAAC,IAAI,CAAC,GAAG,EAAE,OAAO,EAAE;gBAC9C,OAAO,EAAE;oBACP,GAAG,WAAW;oBACd,cAAc,EAAE,kBAAkB;iBACnC;gBACD,OAAO,EAAE,KAAK,CAAC,2CAA2C;aAC3D,CAAC,CAAC;YAEH,OAAO,QAAQ,CAAC,IAAI,CAAC;QACvB,CAAC;QAAC,OAAO,KAAU,EAAE,CAAC;YACpB,IAAI,CAAC,GAAG,CAAC,+BAA+B,KAAK,EAAE,EAAE,OAAO,CAAC,CAAC;YAC1D,MAAM,IAAI,KAAK,CAAC,+BAA+B,KAAK,CAAC,OAAO,EAAE,CAAC,CAAC;QAClE,CAAC;IACH,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,4BAA4B,CAAC,KAAc;QAC/C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,iBAAiB,CAAC,CAAC,EAAE,iBAAiB,CAAC,eAAe,EAAE,KAAK,CAAC,CAAC;YAC1E,IAAI,CAAC,GAAG,CAAC,8CAA8C,KAAK,CAAC,CAAC,CAAC,UAAU,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QAC5F,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,GAAG,CAAC,oDAAoD,KAAK,EAAE,EAAE,OAAO,CAAC,CAAC;YAC/E,gEAAgE;QAClE,CAAC;IACH,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,oBAAoB,CAAC,KAAc;QACvC,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,iBAAiB,CAAC,CAAC,EAAE,iBAAiB,CAAC,WAAW,EAAE,KAAK,CAAC,CAAC;YACtE,IAAI,CAAC,GAAG,CAAC,sCAAsC,KAAK,CAAC,CAAC,CAAC,UAAU,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QACpF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,GAAG,CAAC,4CAA4C,KAAK,EAAE,EAAE,OAAO,CAAC,CAAC;YACvE,gEAAgE;QAClE,CAAC;IACH,CAAC;CACF;AA/GD,gDA+GC"}

package/dist/execution-service.d.ts

@@ -1,22 +1,26 @@
 import { PlaywrightExecutionRequest, PlaywrightExecutionResponse, ScriptExecutionRequest, ScriptExecutionResponse } from './types';
 import { AuthConfig } from './auth-config';
+import { LLMProvider } from './llm-provider';
+import { ProgressReporter } from './progress-reporter';
 /**
  * Service for orchestrating Playwright script execution
  */
 export declare class ExecutionService {
     private playwrightService;
     private llmFacade;
+    private llmProvider;
+    private progressReporter?;
     private creditUsageService;
     private maxConcurrentExecutions;
     private activeExecutions;
     private logger?;
-    constructor(authConfig?: AuthConfig, backendUrl?: string, maxConcurrentExecutions?: number);
+    constructor(authConfig?: AuthConfig, backendUrl?: string, maxConcurrentExecutions?: number, llmProvider?: LLMProvider, progressReporter?: ProgressReporter);
     /**
      * Set a logger callback for capturing execution logs
      */
     setLogger(logger: (message: string, level?: 'log' | 'error' | 'warn') => void): void;
     /**
-     * Log a message using the configured logger or console
+     * Log a message using the configured logger
      */
     private log;
     /**
@@ -25,6 +29,7 @@ export declare class ExecutionService {
     initialize(): Promise<void>;
     /**
      * Set authentication configuration for the service
+     * Note: This recreates the LLM provider with new auth config
      */
     setAuthConfig(authConfig: AuthConfig): void;
     /**

package/dist/execution-service.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"execution-service.d.ts","sourceRoot":"","sources":["../src/execution-service.ts"],"names":[],"mappings":"AACA,OAAO,EACL,0BAA0B,EAC1B,2BAA2B,EAE3B,sBAAsB,EACtB,uBAAuB,EAKxB,MAAM,SAAS,CAAC;AAKjB,OAAO,EAAE,UAAU,EAAE,MAAM,eAAe,CAAC;AAI3C;;GAEG;AACH,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,iBAAiB,CAAoB;IAC7C,OAAO,CAAC,SAAS,CAAY;IAC7B,OAAO,CAAC,kBAAkB,CAAqB;IAC/C,OAAO,CAAC,uBAAuB,CAAS;IACxC,OAAO,CAAC,gBAAgB,CAAgC;IACxD,OAAO,CAAC,MAAM,CAAC,CAA8D;gBAEjE,UAAU,CAAC,EAAE,UAAU,EAAE,UAAU,CAAC,EAAE,MAAM,EAAE,uBAAuB,GAAE,MAAW;IAO9F;;OAEG;IACH,SAAS,CAAC,MAAM,EAAE,CAAC,OAAO,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,KAAK,GAAG,OAAO,GAAG,MAAM,KAAK,IAAI,GAAG,IAAI;IAIpF;;OAEG;IACH,OAAO,CAAC,GAAG;IAcX;;OAEG;IACG,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IAIjC;;OAEG;IACH,aAAa,CAAC,UAAU,EAAE,UAAU,GAAG,IAAI;IAM3C;;OAEG;IACG,aAAa,CAAC,OAAO,EAAE,sBAAsB,GAAG,OAAO,CAAC,uBAAuB,CAAC;IAkBtF;;OAEG;YACW,qBAAqB;IAmBnC;;OAEG;IACG,gBAAgB,CAAC,OAAO,EAAE,0BAA0B,GAAG,OAAO,CAAC,2BAA2B,CAAC;IAgCjG;;OAEG;IACH,OAAO,CAAC,qBAAqB;IAsC7B;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAI5B;;OAEG;IACH,OAAO,IAAI,OAAO;YAIJ,UAAU;YAwDV,eAAe;YAsGf,oBAAoB;IAgBlC,OAAO,CAAC,4BAA4B;YA8CtB,iBAAiB;YAuOjB,eAAe;IAyB7B;;OAEG;IACH,OAAO,CAAC,aAAa;YAeP,oBAAoB;YAoDpB,oBAAoB;YAiBpB,mBAAmB;IAejC,OAAO,CAAC,mBAAmB;IA4B3B,OAAO,CAAC,yBAAyB;YA0CnB,0BAA0B;YAkF1B,iBAAiB;IA0D/B,OAAO,CAAC,qBAAqB;IAsB7B;;OAEG;YACW,iBAAiB;IAI/B;;OAEG;IACH,OAAO,CAAC,kBAAkB;CAkC3B"}
+{"version":3,"file":"execution-service.d.ts","sourceRoot":"","sources":["../src/execution-service.ts"],"names":[],"mappings":"AACA,OAAO,EACL,0BAA0B,EAC1B,2BAA2B,EAE3B,sBAAsB,EACtB,uBAAuB,EAKxB,MAAM,SAAS,CAAC;AAKjB,OAAO,EAAE,UAAU,EAAE,MAAM,eAAe,CAAC;AAI3C,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC7C,OAAO,EAAE,gBAAgB,EAAE,MAAM,qBAAqB,CAAC;AAGvD;;GAEG;AACH,qBAAa,gBAAgB;IAC3B,OAAO,CAAC,iBAAiB,CAAoB;IAC7C,OAAO,CAAC,SAAS,CAAY;IAC7B,OAAO,CAAC,WAAW,CAAc;IACjC,OAAO,CAAC,gBAAgB,CAAC,CAAmB;IAC5C,OAAO,CAAC,kBAAkB,CAAqB;IAC/C,OAAO,CAAC,uBAAuB,CAAS;IACxC,OAAO,CAAC,gBAAgB,CAAgC;IACxD,OAAO,CAAC,MAAM,CAAC,CAA8D;gBAG3E,UAAU,CAAC,EAAE,UAAU,EACvB,UAAU,CAAC,EAAE,MAAM,EACnB,uBAAuB,GAAE,MAAW,EACpC,WAAW,CAAC,EAAE,WAAW,EACzB,gBAAgB,CAAC,EAAE,gBAAgB;IAarC;;OAEG;IACH,SAAS,CAAC,MAAM,EAAE,CAAC,OAAO,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,KAAK,GAAG,OAAO,GAAG,MAAM,KAAK,IAAI,GAAG,IAAI;IAIpF;;OAEG;IACH,OAAO,CAAC,GAAG;IAOX;;OAEG;IACG,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;IAIjC;;;OAGG;IACH,aAAa,CAAC,UAAU,EAAE,UAAU,GAAG,IAAI;IAQ3C;;OAEG;IACG,aAAa,CAAC,OAAO,EAAE,sBAAsB,GAAG,OAAO,CAAC,uBAAuB,CAAC;IAkBtF;;OAEG;YACW,qBAAqB;IAmBnC;;OAEG;IACG,gBAAgB,CAAC,OAAO,EAAE,0BAA0B,GAAG,OAAO,CAAC,2BAA2B,CAAC;IAgCjG;;OAEG;IACH,OAAO,CAAC,qBAAqB;IAsC7B;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAI5B;;OAEG;IACH,OAAO,IAAI,OAAO;YAIJ,UAAU;YAwDV,eAAe;YAsGf,oBAAoB;IAgBlC,OAAO,CAAC,4BAA4B;YA8CtB,iBAAiB;YAqUjB,eAAe;IA0B7B;;OAEG;IACH,OAAO,CAAC,aAAa;YAeP,oBAAoB;YAoDpB,oBAAoB;YAiBpB,mBAAmB;IAcjC,OAAO,CAAC,mBAAmB;IA4B3B,OAAO,CAAC,yBAAyB;YA0CnB,0BAA0B;YAkF1B,iBAAiB;IA0D/B,OAAO,CAAC,qBAAqB;IAyB7B;;OAEG;YACW,iBAAiB;IAI/B;;OAEG;IACH,OAAO,CAAC,kBAAkB;CAkC3B"}