testchimp-runner-core 0.0.35 → 0.0.36

package/releasenotes/RELEASE_0.0.28.md

@@ -1,286 +0,0 @@
-# Runner-Core 0.0.28 Release - Scriptservice Integration
-
-## Overview
-
-Extended runner-core with lifecycle callbacks and created SmartTestRunnerCoreV2 wrapper to enable scriptservice integration. This release makes runner-core a drop-in replacement for scriptservice's duplicated SmartTestRunnerCore.
-
-## Changes
-
-### 1. Extended ProgressReporter with Lifecycle Callbacks
-
-**File:** `src/progress-reporter.ts`
-
-Added optional lifecycle callbacks to ProgressReporter interface:
-
-```typescript
-export interface StepInfo {
-  stepId?: string;
-  stepNumber: number;
-  description: string;
-  code?: string;
-}
-
-export interface ProgressReporter {
-  // ... existing callbacks ...
-  
-  // NEW: Lifecycle callbacks (used by scriptservice, ignored by local clients)
-  beforeStartTest?(page: any, browser: any, context: any): Promise<void>;
-  beforeStepStart?(step: StepInfo, page: any): Promise<void>;
-  afterEndTest?(status: 'passed' | 'failed', error?: string, page?: any): Promise<void>;
-}
-```
-
-**Purpose:**
-- `beforeStartTest`: Initialize browser context, set up DB records (scriptservice only)
-- `beforeStepStart`: Update step status to IN_PROGRESS in DB (scriptservice only)
-- `afterEndTest`: Write final status to DB, cleanup resources (scriptservice only)
-- Local clients (vs-ext, github-action) can ignore these - they use return values
-
-### 2. Integrated Lifecycle Callbacks
-
-**Files:** 
-- `src/execution-service.ts` - RUN_EXACTLY and AI_REPAIR modes
-- `src/scenario-worker-class.ts` - Orchestrator mode
-
-**Integration Points:**
-
-**ExecutionService (RUN_EXACTLY):**
-```typescript
-// Before script execution
-if (this.progressReporter?.beforeStartTest) {
-  await this.progressReporter.beforeStartTest(page, browser, context);
-}
-
-// After execution (success or failure)
-if (this.progressReporter?.afterEndTest) {
-  await this.progressReporter.afterEndTest(
-    success ? 'passed' : 'failed',
-    error,
-    page
-  );
-}
-```
-
-**ExecutionService (AI_REPAIR):**
-```typescript
-// Before each step
-if (this.progressReporter?.beforeStepStart) {
-  await this.progressReporter.beforeStepStart(
-    { stepNumber, description, code },
-    page
-  );
-}
-```
-
-**ScenarioWorker (Orchestrator):**
-```typescript
-// After browser initialization
-if (this.progressReporter?.beforeStartTest) {
-  await this.progressReporter.beforeStartTest(page, browser, context);
-}
-
-// Before each orchestrator step
-if (this.progressReporter?.beforeStepStart) {
-  await this.progressReporter.beforeStepStart({ stepNumber, description }, page);
-}
-
-// In finally block (before browser close)
-if (this.progressReporter?.afterEndTest) {
-  await this.progressReporter.afterEndTest(
-    overallSuccess ? 'passed' : 'failed',
-    error,
-    page
-  );
-}
-```
-
-### 3. Version Bump
-
-**File:** `package.json`
-
-Updated version from `0.0.27` to `0.0.28`
-
-## Scriptservice Integration
-
-### 1. Created ScriptserviceLLMProvider
-
-**File:** `services/scriptservice/providers/scriptservice-llm-provider.ts` (NEW)
-
-Implements `LLMProvider` interface for scriptservice:
-- Wraps scriptservice's existing OpenAI client
-- Supports both text and vision prompts
-- Uses scriptservice's OpenAI API key from ConfigService
-- No backend proxy - all calls are local
-- No token tracking needed (scriptservice doesn't use this interface for tracking)
-
-### 2. Created SmartTestRunnerCoreV2
-
-**File:** `services/scriptservice/smart-test-runner-core-v2.ts` (NEW)
-
-Drop-in replacement for SmartTestRunnerCore:
-- **Same Interface:** Maintains exact same constructor and methods (runExactly, runWithRepair)
-- **Uses runner-core:** Delegates to TestChimpService internally
-- **Callback Mapping:** Maps scriptservice callbacks to runner-core ProgressReporter
-- **Lifecycle Support:** Properly calls beforeStartTest, beforeStepStart, afterEndTest
-- **No Breaking Changes:** Can replace SmartTestRunnerCore with zero code changes
-
-**Key Features:**
-```typescript
-export class SmartTestRunnerCoreV2 {
-  constructor(config: RunnerConfig) {
-    // Create local LLM provider (no backend)
-    const llmProvider = new ScriptserviceLLMProvider();
-    
-    // Map callbacks to progress reporter
-    const progressReporter: ProgressReporter = {
-      onStepProgress: async (stepProgress) => {
-        // Call scriptservice's onStepComplete
-      },
-      beforeStartTest: config.callbacks?.beforeStartTest,
-      beforeStepStart: config.callbacks?.beforeStepStart,
-      afterEndTest: config.callbacks?.afterEndTest
-    };
-    
-    // Initialize runner-core (no auth, no backend)
-    this.runnerCore = new TestChimpService(
-      undefined, // No file handler
-      undefined, // No auth
-      undefined, // No backend URL
-      1,         // Single worker
-      llmProvider,
-      progressReporter
-    );
-  }
-  
-  async runExactly(script: string): Promise<RunExactlyResult> { /* ... */ }
-  async runWithRepair(steps: TestStepWithId[]): Promise<RunWithRepairResult> { /* ... */ }
-}
-```
-
-### 3. Updated Scriptservice Consumers
-
-**Files Updated:**
-- `services/scriptservice/smart-test-execution-handler.ts`
-- `services/scriptservice/workers/test-based-explorer.ts`
-- `services/scriptservice/package.json` - Added `testchimp-runner-core: ^0.0.28`
-
-**Migration Strategy:**
-```typescript
-// Environment flag for gradual rollout
-const useV2 = process.env.USE_RUNNER_CORE_V2 !== 'false'; // Default: true
-const runner = useV2 
-  ? new SmartTestRunnerCoreV2(runnerConfig) 
-  : new SmartTestRunnerCore(runnerConfig);
-```
-
-**Rollout Plan:**
-1. Deploy with `USE_RUNNER_CORE_V2=true` (default)
-2. Monitor scriptservice execution logs
-3. If issues occur, set `USE_RUNNER_CORE_V2=false` to rollback
-4. Once validated, remove flag and delete old SmartTestRunnerCore
-
-## Benefits
-
-### For Scriptservice:
-1. **Zero Axios Calls:** All LLM calls local, all DB writes via callbacks
-2. **Auto-Updates:** Automatically get runner-core improvements (better prompts, vision, orchestrator)
-3. **Code Deduplication:** Remove ~600 lines of duplicated execution/repair logic
-4. **Consistency:** Same prompts and logic as vs-extension and github-action
-
-### For Runner-Core:
-1. **Universal Library:** Works for both client-side (vs-ext) and server-side (scriptservice)
-2. **Flexible Architecture:** Callbacks allow different execution patterns
-3. **Backward Compatible:** Existing consumers unaffected (callbacks are optional)
-
-## Testing
-
-### Scriptservice Testing:
-```bash
-# Test with V2 (default)
-USE_RUNNER_CORE_V2=true npm start
-
-# Test with V1 (legacy fallback)
-USE_RUNNER_CORE_V2=false npm start
-```
-
-### Expected Behavior:
-- V2: Logs show "using SmartTestRunnerCoreV2 (runner-core)"
-- V1: Logs show "using SmartTestRunnerCore (legacy)"
-- All callbacks (beforeStartTest, beforeStepStart, afterEndTest, onStepComplete) should fire
-- DB writes should happen via callbacks
-- Screenshots should upload to GCS
-- Journey reporting should work
-
-## Publishing
-
-### Prerequisites:
-1. ✅ Build runner-core: `npm run build`
-2. ✅ No lint errors
-3. ✅ Version bumped to 0.0.28
-4. ⏸️ **Awaiting user permission to publish to npm**
-
-### Publish Command (when approved):
-```bash
-cd /Users/nuwansam/IdeaProjects/AwareRepo/local/runner-core
-npm publish
-```
-
-### Post-Publish:
-1. Install in scriptservice: `npm install testchimp-runner-core@0.0.28`
-2. Install in vs-ext: Update to 0.0.28
-3. Install in github-action: Update to 0.0.28
-
-## Migration Checklist
-
-### Phase 1: Deploy with V2 (Default)
-- [x] Add lifecycle callbacks to ProgressReporter
-- [x] Integrate callbacks in ExecutionService and ScenarioService
-- [x] Create ScriptserviceLLMProvider
-- [x] Create SmartTestRunnerCoreV2
-- [x] Update scriptservice consumers with environment flag
-- [ ] Publish runner-core 0.0.28 (needs user permission)
-- [ ] Install 0.0.28 in scriptservice
-- [ ] Deploy scriptservice with USE_RUNNER_CORE_V2=true
-- [ ] Monitor logs and execution results
-
-### Phase 2: Validate (1-2 weeks)
-- [ ] Verify all smart test executions work
-- [ ] Verify explorer mode works
-- [ ] Verify script generation works
-- [ ] Check DB writes happen correctly
-- [ ] Check screenshot uploads work
-- [ ] Monitor for any errors or regressions
-
-### Phase 3: Full Migration
-- [ ] Remove environment flag check
-- [ ] Replace all `new SmartTestRunnerCore` with `new SmartTestRunnerCoreV2`
-- [ ] Delete old `smart-test-runner-core.ts` file
-- [ ] Optional: Rename V2 to just `SmartTestRunnerCore`
-
-## Breaking Changes
-
-None. All changes are backward compatible.
-
-## Files Changed
-
-### Runner-Core:
-- `src/progress-reporter.ts` - Added lifecycle callbacks
-- `src/execution-service.ts` - Integrated callbacks
-- `src/scenario-worker-class.ts` - Integrated callbacks
-- `package.json` - Version bump to 0.0.28
-
-### Scriptservice:
-- `providers/scriptservice-llm-provider.ts` - NEW
-- `smart-test-runner-core-v2.ts` - NEW
-- `smart-test-execution-handler.ts` - Added V2 usage with flag
-- `workers/test-based-explorer.ts` - Added V2 usage with flag
-- `package.json` - Added testchimp-runner-core dependency
-
-## Notes
-
-- Lifecycle callbacks are **optional** - local clients (vs-ext, github-action) don't need them
-- Scriptservice uses callbacks for DB writes and resource management
-- V2 wrapper uses existing page/browser/context (doesn't create new ones)
-- LLM calls in V2 are all local (no backend proxy)
-- No authentication needed for scriptservice (already authenticated at service level)
-

package/src/auth-config.ts

@@ -1,84 +0,0 @@
-/**
- * Authentication configuration for TestChimp services
- * Supports both user PAT and project API key authentication modes
- */
-
-export interface UserPATAuthConfig {
-  mode: 'user_pat';
-  userAuthKey: string;
-  userMail: string;
-}
-
-export interface ProjectApiKeyAuthConfig {
-  mode: 'project_api_key';
-  apiKey: string;
-  projectId: string;
-}
-
-export type AuthConfig = UserPATAuthConfig | ProjectApiKeyAuthConfig;
-
-/**
- * Helper function to create user PAT auth configuration
- */
-export function createUserPATAuth(userAuthKey: string, userMail: string): UserPATAuthConfig {
-  return {
-    mode: 'user_pat',
-    userAuthKey,
-    userMail
-  };
-}
-
-/**
- * Helper function to create project API key auth configuration
- */
-export function createProjectApiKeyAuth(apiKey: string, projectId: string): ProjectApiKeyAuthConfig {
-  return {
-    mode: 'project_api_key',
-    apiKey,
-    projectId
-  };
-}
-
-/**
- * Helper function to create auth configuration from environment variables
- * Supports both authentication modes based on available environment variables
- */
-export function createAuthConfigFromEnv(): AuthConfig | null {
-  // Check for project API key authentication first (for CI/CD)
-  const apiKey = process.env.TESTCHIMP_API_KEY;
-  const projectId = process.env.TESTCHIMP_PROJECT_ID;
-  
-  if (apiKey && projectId) {
-    return createProjectApiKeyAuth(apiKey, projectId);
-  }
-  
-  // Fall back to user PAT authentication (for VS Code extension)
-  const userAuthKey = process.env.TESTCHIMP_USER_AUTH_KEY;
-  const userMail = process.env.TESTCHIMP_USER_MAIL;
-  
-  if (userAuthKey && userMail) {
-    return createUserPATAuth(userAuthKey, userMail);
-  }
-  
-  return null;
-}
-
-/**
- * Helper function to get authentication headers for HTTP requests
- */
-export function getAuthHeaders(authConfig: AuthConfig): Record<string, string> {
-  switch (authConfig.mode) {
-    case 'user_pat':
-      return {
-        'user_auth_key': authConfig.userAuthKey,
-        'user_mail': authConfig.userMail
-      };
-    case 'project_api_key':
-      return {
-        'TestChimp-Api-Key': authConfig.apiKey,
-        'project-id': authConfig.projectId
-      };
-    default:
-      throw new Error('Invalid authentication configuration');
-  }
-}

package/src/credit-usage-service.ts

@@ -1,188 +0,0 @@
-import axios from 'axios';
-import { AuthConfig, getAuthHeaders } from './auth-config';
-import { loadEnvConfig } from './env-loader';
-
-export enum CreditUsageReason {
-  UNKNOWN_CREDIT_USAGE_REASON = 0,
-  EXPLORER_STEP = 1,
-  TEST_REPAIR = 2,
-  SCRIPT_GENERATE = 3
-}
-
-export interface InsertCreditUsageRequest {
-  credits?: number;
-  usage_reason?: CreditUsageReason;
-  job_id?: string;
-}
-
-export interface InsertCreditUsageResponse {
-  // Empty response as per the protobuf definition
-}
-
-/**
- * Credit usage information for callback
- */
-export interface CreditUsage {
-  credits: number;
-  usageReason: CreditUsageReason;
-  jobId?: string;
-  timestamp: number;
-}
-
-/**
- * Callback for credit usage reporting
- * Allows consumers to track credit usage in their own systems
- */
-export type CreditUsageCallback = (usage: CreditUsage) => void | Promise<void>;
-
-export class CreditUsageService {
-  private backendUrl: string;
-  private authConfig: AuthConfig | null;
-  private logger?: (message: string, level?: 'log' | 'error' | 'warn') => void;
-  private creditUsageCallback?: CreditUsageCallback;
-
-  constructor(authConfig?: AuthConfig, backendUrl?: string, creditUsageCallback?: CreditUsageCallback) {
-    // Use provided backend URL or fall back to environment configuration
-    if (backendUrl) {
-      this.backendUrl = backendUrl;
-    } else {
-      // Fall back to environment configuration for backward compatibility
-      const envConfig = loadEnvConfig();
-      this.backendUrl = envConfig.TESTCHIMP_BACKEND_URL;
-    }
-    
-    this.authConfig = authConfig || null;
-    this.creditUsageCallback = creditUsageCallback;
-  }
-
-  /**
-   * Set a logger callback for capturing execution logs
-   */
-  setLogger(logger: (message: string, level?: 'log' | 'error' | 'warn') => void): void {
-    this.logger = logger;
-  }
-
-  /**
-   * Set credit usage callback
-   * Allows consumers to track credit usage in their own systems
-   */
-  setCreditUsageCallback(callback: CreditUsageCallback): void {
-    this.creditUsageCallback = callback;
-  }
-
-  /**
-   * Log a message using the configured logger
-   */
-  private log(message: string, level: 'log' | 'error' | 'warn' = 'log'): void {
-    if (this.logger) {
-      this.logger(message, level);
-    }
-    // No console fallback - logs are routed to consumer
-  }
-
-  /**
-   * Update authentication configuration
-   */
-  setAuthConfig(authConfig: AuthConfig): void {
-    this.authConfig = authConfig;
-  }
-
-  /**
-   * Get current authentication configuration
-   */
-  getAuthConfig(): AuthConfig | null {
-    return this.authConfig;
-  }
-
-  /**
-   * Report credit usage via callback and/or backend
-   * 
-   * Behavior:
-   * 1. If callback provided: Call callback (server-side uses this to update DB directly)
-   * 2. If NO callback but auth configured: Make axios call to backend (client-side: vs-ext, github action)
-   * 3. Server-side: Provides callback, no axios calls made
-   * 4. Client-side: No callback, uses auth for axios calls to backend
-   */
-  async reportCreditUsage(
-    credits: number = 1,
-    usageReason: CreditUsageReason,
-    jobId?: string
-  ): Promise<InsertCreditUsageResponse> {
-    const creditUsage: CreditUsage = {
-      credits,
-      usageReason,
-      jobId,
-      timestamp: Date.now()
-    };
-    
-    // 1. If callback provided: Use callback (server-side path)
-    if (this.creditUsageCallback) {
-      try {
-        await this.creditUsageCallback(creditUsage);
-        this.log(`Credit usage reported via callback: ${credits} credits for ${CreditUsageReason[usageReason]}`);
-        return {}; // Callback succeeded, no need for backend call
-      } catch (error: any) {
-        this.log(`Credit usage callback failed: ${error.message}`, 'error');
-        throw error; // Callback failure is critical for server-side
-      }
-    }
-    
-    // 2. No callback: Make axios call if auth configured (client-side path: vs-ext, github action)
-    if (this.authConfig) {
-      try {
-        const authHeaders = getAuthHeaders(this.authConfig);
-        const url = `${this.backendUrl}/localagent/insert_credit_usage`;
-        
-        const request: InsertCreditUsageRequest = {
-          credits,
-          usage_reason: usageReason,
-          job_id: jobId
-        };
-
-        const response = await axios.post(url, request, {
-          headers: {
-            ...authHeaders,
-            'Content-Type': 'application/json'
-          },
-          timeout: 10000 // 10 second timeout for credit usage calls
-        });
-
-        this.log(`Credit usage reported to backend: ${credits} credits for ${CreditUsageReason[usageReason]}`);
-        return response.data;
-      } catch (error: any) {
-        this.log(`Credit usage backend report failed: ${error.message}`, 'error');
-        throw error; // Backend failure is critical for client-side
-      }
-    }
-    
-    // 3. No callback and no auth - no credit tracking (development mode)
-    this.log(`Credit usage not tracked (no callback or auth configured): ${credits} credits for ${CreditUsageReason[usageReason]}`, 'warn');
-    return {};
-  }
-
-  /**
-   * Report script generation credit usage
-   */
-  async reportScriptGenerationCredit(jobId?: string): Promise<void> {
-    try {
-      await this.reportCreditUsage(1, CreditUsageReason.SCRIPT_GENERATE, jobId);
-      this.log(`Credit usage reported for script generation${jobId ? ` (job: ${jobId})` : ''}`);
-    } catch (error) {
-      this.log(`Failed to report script generation credit usage: ${error}`, 'error');
-      // Don't throw - credit reporting should not break the main flow
-    }
-  }
-
-  /**
-   * Report AI repair credit usage
-   */
-  async reportAIRepairCredit(jobId?: string): Promise<void> {
-    try {
-      await this.reportCreditUsage(1, CreditUsageReason.TEST_REPAIR, jobId);
-      this.log(`Credit usage reported for AI repair${jobId ? ` (job: ${jobId})` : ''}`);
-    } catch (error) {
-      this.log(`Failed to report AI repair credit usage: ${error}`, 'error');
-      // Don't throw - credit reporting should not break the main flow
-    }
-  }
-}

package/src/env-loader.ts

@@ -1,103 +0,0 @@
-/**
- * Environment Configuration Loader
- * 
- * This module loads environment configuration from packaged env files
- * instead of relying on process.env which may not be available in VS Code extensions
- */
-
-interface EnvConfig {
-  TESTCHIMP_BACKEND_URL: string;
-}
-
-let cachedConfig: EnvConfig | null = null;
-
-/**
- * Load environment configuration from the packaged env file
- * Falls back to process.env if env file is not available
- */
-export function loadEnvConfig(): EnvConfig {
-  if (cachedConfig) {
-    return cachedConfig;
-  }
-
-  try {
-    // Try multiple possible paths for the env file
-    const possiblePaths = [
-      require('path').join(String(__dirname), 'env'),
-      require('path').join(String(__dirname), './env'),
-      require('path').join(String(__dirname), '../env'),
-      require('path').join(String(__dirname), '../../runner-core/env'),
-      require('path').join(String(__dirname), '../../runner-core/dist/env'),
-      require('path').join(String(process.cwd()), 'env'),
-      require('path').join(String(process.cwd()), 'runner-core/env'),
-      require('path').join(String(process.cwd()), 'runner-core/dist/env'),
-      require('path').join(String(process.cwd()), 'local/runner-core/env'),
-      require('path').join(String(process.cwd()), 'local/runner-core/dist/env')
-    ];
-    
-    let envContent = '';
-    let envPath = '';
-    
-    for (const path of possiblePaths) {
-      try {
-        envContent = require('fs').readFileSync(path, 'utf8');
-        envPath = path;
-        break;
-      } catch (error) {
-        // Silently continue to next path
-      }
-    }
-    
-    if (!envContent) {
-      throw new Error('Could not find env file in any of the expected locations');
-    }
-    
-    const config: Partial<EnvConfig> = {};
-    
-    envContent.split('\n').forEach((line: string) => {
-      const trimmedLine = line.trim();
-      if (trimmedLine && !trimmedLine.startsWith('#')) {
-        const [key, value] = trimmedLine.split('=');
-        if (key && value) {
-          const trimmedKey = key.trim() as keyof EnvConfig;
-          config[trimmedKey] = value.trim();
-        }
-      }
-    });
-    
-    // Ensure required properties are present
-    const finalConfig: EnvConfig = {
-      TESTCHIMP_BACKEND_URL: config.TESTCHIMP_BACKEND_URL || 'https://featureservice.testchimp.io'
-    };
-    
-    cachedConfig = finalConfig;
-    return finalConfig;
-  } catch (error) {
-    // Fallback to process.env
-    let backendUrl = process.env.TESTCHIMP_BACKEND_URL;
-    
-    // If no explicit backend URL, determine from TESTCHIMP_ENV
-    if (!backendUrl) {
-      const testchimpEnv = process.env.TESTCHIMP_ENV;
-      if (testchimpEnv === 'staging') {
-        backendUrl = 'https://featureservice-staging.testchimp.io';
-      } else {
-        backendUrl = 'https://featureservice.testchimp.io';
-      }
-    }
-    
-    const config: EnvConfig = {
-      TESTCHIMP_BACKEND_URL: backendUrl
-    };
-    cachedConfig = config;
-    return config;
-  }
-}
-
-/**
- * Get a specific environment variable
- */
-export function getEnvVar(key: keyof EnvConfig): string | undefined {
-  const config = loadEnvConfig();
-  return config[key];
-}