testchimp-runner-core 0.0.34 → 0.0.36

package/plandocs/COORDINATE_MODE_DIAGNOSIS.md

@@ -1,144 +0,0 @@
-# Coordinate Mode Diagnosis - Live Test Results
-
-## Test Scenario: PeopleHR Employee Information Flow
-
-### ✅ What Worked:
-
-1. **Coordinate fallback DID activate** (after fix from >= 3 to >= 5)
-2. **Agent successfully used coordinates** at (87.5%, 23.438%)
-3. **Physical clicks succeeded** - page.mouse.click(1120, 169)
-4. **Agent learned** to stick with coordinates after selectors failed
-
-### ❌ What Didn't Work:
-
-**Agent hit max iterations (8) without marking "complete"**
-
-## Detailed Step 6 Flow:
-
-```
-Iteration 1: Selector attempt → Timeout ❌
-Iteration 2: Selector attempt → Timeout ❌
-Iteration 3: Selector attempt → Timeout ❌
-Iteration 4: 🎯 COORDINATE MODE → Click (87.5%, 23.438%) → ✅ Success
-Iteration 5: Repeat coordinate → ✅ Success
-Iteration 6: Repeat coordinate → ✅ Success (?)
-Iteration 7: Repeat coordinate → ✅ Success
-Iteration 8: Repeat coordinate → ✅ Success
-Result: ⚠️ Max iterations → system_limit
-```
-
-## Root Cause Analysis:
-
-### Problem: **No Goal Verification After Coordinate Success**
-
-**With selectors:**
-```typescript
-await page.getByRole('button').click();
-// Can verify: await expect(button).toHaveState('pressed')
-// Can check: New elements appeared, URL changed, etc.
-```
-
-**With coordinates:**
-```typescript
-await page.mouse.click(1120, 169);
-// ❓ Did it work? No element reference!
-// ❓ How to verify? Can't check button state
-// ❓ What changed? Need to inspect DOM/screenshot
-```
-
-### Why Agent Kept Retrying:
-
-**Agent's reasoning (iterations 5-8):**
-- "Coordinate click succeeded (executed without error)"
-- "But I don't know if goal was achieved"
-- "Step says 'Click on New' - did the New form open?"
-- "I should try again to be sure..."
-- → **Loops until max iterations**
-
-## Solutions to Consider:
-
-### Option 1: **Trust Coordinate Success** (Simple)
-After coordinate click succeeds:
-- Wait 500ms for UI response
-- Mark status="complete" automatically
-- Assume click worked (trust the coordinates)
-
-```typescript
-if (coordinateAction && coordResult.allSucceeded) {
-  await page.waitForTimeout(500); // Let UI respond
-  return { status: 'complete', reasoning: 'Coordinate click succeeded' };
-}
-```
-
-**Pros**: Simple, fast
-**Cons**: No verification of actual goal achievement
-
-### Option 2: **Visual Verification** (Better)
-After coordinate click:
-- Wait 500ms
-- Take screenshot
-- Compare before/after
-- If changed → complete, else → retry with different coords
-
-```typescript
-const beforeScreenshot = await page.screenshot();
-await page.mouse.click(x, y);
-await page.waitForTimeout(500);
-const afterScreenshot = await page.screenshot();
-if (screenshotsAreDifferent(before, after)) {
-  return { status: 'complete' };
-}
-```
-
-**Pros**: Validates something changed
-**Cons**: Slower, more LLM calls
-
-### Option 3: **DOM Change Detection** (Balanced)
-After coordinate click:
-- Capture DOM snapshot before
-- Click coordinates
-- Capture DOM snapshot after
-- If new elements/navigation → complete
-
-```typescript
-const beforeUrl = page.url();
-const beforeElements = await getEnhancedPageInfo(page);
-await page.mouse.click(x, y);
-await page.waitForTimeout(500);
-const afterUrl = page.url();
-const afterElements = await getEnhancedPageInfo(page);
-
-if (afterUrl !== beforeUrl || afterElements.count !== beforeElements.count) {
-  return { status: 'complete', reasoning: 'Page state changed after coordinate click' };
-}
-```
-
-**Pros**: Fast, objective verification
-**Cons**: Might miss subtle changes (modal opens without URL/element count change)
-
-### Option 4: **Prompt Guidance** (Immediate)
-Update prompt to tell agent:
-"After coordinate click succeeds, mark status='complete' unless you can clearly verify it failed"
-
-**Pros**: No code changes
-**Cons**: Relies on LLM judgment
-
-## Recommendation:
-
-**Hybrid approach:**
-1. **Immediate** (Prompt): Tell agent to trust coordinate success
-2. **Phase 2** (Code): Add DOM change detection for validation
-
-## Current Status:
-
-- ✅ Coordinate fallback works technically
-- ✅ Physical clicks succeed
-- ❌ Agent doesn't know when to stop
-- 🔧 Need completion detection logic
-
-## Test Results Summary:
-
-**Steps 1-5**: ✅ All completed successfully
-**Step 6**: ⚠️ Coordinates worked but hit max iterations (no completion detection)
-**Overall**: Coordinate mode is functional but needs completion logic
-

package/plandocs/CREDIT_CALLBACK_ARCHITECTURE.md

@@ -1,253 +0,0 @@
-# Credit Usage Callback Architecture
-
-## Summary
-Added callback-based credit usage reporting to allow server-side integration to update DB directly without axios calls, while client-side continues using axios calls to backend API.
-
-## Architecture
-
-### Callback-First Approach
-
-```typescript
-export interface CreditUsage {
-  credits: number;
-  usageReason: CreditUsageReason;
-  jobId?: string;
-  timestamp: number;
-}
-
-export type CreditUsageCallback = (usage: CreditUsage) => void | Promise<void>;
-```
-
-### Behavior
-
-```typescript
-async reportCreditUsage(credits, usageReason, jobId) {
-  const creditUsage = { credits, usageReason, jobId, timestamp: Date.now() };
-  
-  // 1. If callback provided: Use callback (SERVER-SIDE)
-  if (this.creditUsageCallback) {
-    await this.creditUsageCallback(creditUsage);
-    return {}; // No axios call needed
-  }
-  
-  // 2. No callback but auth configured: Use axios (CLIENT-SIDE)
-  if (this.authConfig) {
-    await axios.post(`${backend}/localagent/insert_credit_usage`, ...);
-    return response.data;
-  }
-  
-  // 3. No callback and no auth: Development mode
-  return {};
-}
-```
-
-## Usage Scenarios
-
-### Server-Side (scriptservice)
-
-```typescript
-import { TestChimpService, CreditUsage, CreditUsageReason } from 'testchimp-runner-core';
-
-const service = new TestChimpService(
-  fileHandler,
-  undefined, // NO auth config
-  backendUrl,
-  maxWorkers,
-  llmProvider,
-  progressReporter,
-  orchestratorOptions,
-  async (creditUsage: CreditUsage) => {
-    // Update DB directly - NO axios calls
-    await db.insertCreditUsage({
-      credits: creditUsage.credits,
-      usageReason: creditUsage.usageReason,
-      jobId: creditUsage.jobId,
-      timestamp: creditUsage.timestamp
-    });
-  }
-);
-```
-
-**Result:**
-- ✅ Callback called → DB updated directly
-- ❌ No axios calls (no auth configured)
-- ✅ Full control over DB updates
-
-### Client-Side (vs-extension, github-action)
-
-```typescript
-import { TestChimpService } from 'testchimp-runner-core';
-
-const service = new TestChimpService(
-  fileHandler,
-  authConfig, // Auth configured
-  backendUrl,
-  maxWorkers,
-  llmProvider,
-  progressReporter,
-  orchestratorOptions
-  // NO callback - uses axios
-);
-```
-
-**Result:**
-- ❌ No callback provided
-- ✅ Axios call made to backend API (because auth configured)
-- ✅ Backend handles DB update
-
-### Development Mode (local testing)
-
-```typescript
-const service = new TestChimpService(
-  fileHandler
-  // No auth, no callback
-);
-```
-
-**Result:**
-- ❌ No callback
-- ❌ No auth
-- ⚠️ Warning logged: "Credit usage not tracked"
-- ✅ Continues without error
-
-## API
-
-### Constructor
-```typescript
-new TestChimpService(
-  fileHandler?,
-  authConfig?,
-  backendUrl?,
-  maxWorkers?,
-  llmProvider?,
-  progressReporter?,
-  orchestratorOptions?,
-  creditUsageCallback? // NEW
-)
-```
-
-### Method
-```typescript
-service.setCreditUsageCallback((creditUsage) => {
-  // Handle credit usage in your system
-  console.log(`Used ${creditUsage.credits} credits for ${creditUsage.usageReason}`);
-});
-```
-
-## Exported Types
-
-```typescript
-export {
-  CreditUsageCallback,     // Type for callback function
-  CreditUsage,             // Interface for usage data
-  CreditUsageReason        // Enum: SCRIPT_GENERATE, TEST_REPAIR, etc.
-};
-```
-
-## Benefits
-
-### For Server-Side
-1. **No Network Calls** - Direct DB updates via callback
-2. **Full Control** - Custom logic for credit tracking
-3. **Performance** - No HTTP round-trip overhead
-4. **Reliability** - No network failures
-
-### For Client-Side
-1. **Backward Compatible** - Existing code works unchanged
-2. **Centralized** - Backend API handles all credit logic
-3. **Simple** - Just configure auth, no callback needed
-
-### For Both
-1. **Flexible** - Each consumer decides how to handle credits
-2. **Testable** - Can mock callbacks easily
-3. **Observable** - Callback provides visibility into credit usage
-
-## Implementation Notes
-
-### Preservation Across Service Recreations
-
-Credit callback is stored and reapplied when services are recreated:
-
-```typescript
-// Store in TestChimpService
-private creditUsageCallback?: CreditUsageCallback;
-
-// Pass to CreditUsageService on every recreation
-this.creditUsageService = new CreditUsageService(
-  this.authConfig,
-  this.backendUrl,
-  this.creditUsageCallback  // Always preserved
-);
-```
-
-### Error Handling
-
-**Callback-based (server-side):**
-- Callback error → Throws (critical for DB updates)
-
-**Axios-based (client-side):**
-- Axios error → Throws (critical for credit tracking)
-
-**Development mode:**
-- No tracking → Logs warning, continues
-
-## Example: Server-Side Integration
-
-```typescript
-// In scriptservice
-import { TestChimpService, CreditUsage, CreditUsageReason } from 'testchimp-runner-core';
-
-class ScriptService {
-  private testChimpService: TestChimpService;
-  
-  constructor() {
-    this.testChimpService = new TestChimpService(
-      new CustomFileHandler(),
-      undefined, // No auth - server-side doesn't need it
-      'http://localhost:3000', // Internal backend URL
-      5,
-      new CustomLLMProvider(), // Server has its own LLM provider
-      customProgressReporter,
-      { useOrchestrator: true },
-      async (creditUsage: CreditUsage) => {
-        // Direct DB update - no HTTP calls
-        await this.creditRepository.insert({
-          userId: this.getCurrentUserId(),
-          credits: creditUsage.credits,
-          reason: creditUsage.usageReason,
-          jobId: creditUsage.jobId,
-          timestamp: new Date(creditUsage.timestamp)
-        });
-      }
-    );
-  }
-}
-```
-
-## Files Modified
-
-1. `/src/credit-usage-service.ts` - Added callback support, callback-first logic
-2. `/src/index.ts` - Accept credit callback in constructor, expose `setCreditUsageCallback()`, preserve across recreations
-3. Exported types: `CreditUsage`, `CreditUsageCallback`, `CreditUsageReason`
-
-## Testing
-
-### Server-Side
-1. Set credit callback
-2. Generate script
-3. Verify callback called with correct credit data
-4. Verify NO axios calls made
-
-### Client-Side
-1. Configure auth (no callback)
-2. Generate script
-3. Verify axios call made to backend
-4. Verify backend receives credit data
-
-## Backward Compatibility
-
-✅ Fully backward compatible
-- Existing consumers work unchanged
-- Optional callback parameter
-- Existing axios behavior preserved for client-side
-