testchimp-runner-core 0.0.26 → 0.0.29

package/RELEASE_0.0.26.md

@@ -0,0 +1,165 @@
+# Release 0.0.26 - Fix Orchestrator Reasoning Logs
+
+## Summary
+Fixed bug where orchestrator reasoning logs (💭 REASONING, 🧠 SELF-REFLECTION, etc.) were not appearing in VS Code output channel due to outputChannel being lost during service recreation.
+
+## Bug Fixed
+
+### The Problem
+Orchestrator reasoning logs were only showing in `console.log()` (debug console), not in:
+- ❌ VS Code "TestChimp Agent" output channel
+- ❌ Log files
+
+Users couldn't see what the agent was thinking!
+
+**Root Cause:**
+When `setAuthConfig()` was called, it recreated `ScenarioService`, which lost the `outputChannel` reference. Workers were then created without access to the output channel, causing all orchestrator logs to fall back to `console.log()` only.
+
+**Call sequence that caused the bug:**
+1. `embeddedService.initialize()` → calls `service.setOutputChannel()` ✅
+2. Then calls `service.setAuthConfig()` which:
+   - Recreates `ScenarioService` ❌ outputChannel lost!
+   - Calls `scenarioService.initialize()` → creates workers without outputChannel
+3. Orchestrator logs → `console.log()` only (not visible in output channel)
+
+## Solution
+
+### Changes in runner-core
+
+**TestChimpService (`index.ts`):**
+
+1. **Store outputChannel** in private field:
+   ```typescript
+   private outputChannel?: any;  // Store outputChannel to preserve it across service recreations
+   ```
+
+2. **Preserve it in setOutputChannel()**:
+   ```typescript
+   setOutputChannel(outputChannel: any): void {
+     this.outputChannel = outputChannel;  // Store for future use
+     if (typeof this.scenarioService?.setOutputChannel === 'function') {
+       this.scenarioService.setOutputChannel(outputChannel);
+     }
+   }
+   ```
+
+3. **Reapply after service recreation** (in both `setAuthConfig()` and `setBackendUrl()`):
+   ```typescript
+   // Reapply outputChannel if we have one (critical for orchestrator logs)
+   if (this.outputChannel) {
+     this.scenarioService.setOutputChannel(this.outputChannel);
+   }
+   ```
+
+### Result
+
+Now orchestrator logs flow correctly:
+```
+Orchestrator (runner-core)
+  ↓ this.logger?.()
+ScenarioWorker.log()
+  ↓ outputChannel.appendLine()
+VS Code Output Channel ✅
+  ↓ dual-write
+Log File ✅
+```
+
+## What Now Appears in Output Channel
+
+**All Environments:**
+- ✅ `💭 REASONING: [agent's reasoning]`
+- ✅ `🧠 SELF-REFLECTION: [guidance for next iteration]`
+- ✅ `💡 Why: [command reasoning]`
+- ✅ `🔧 TOOLS: [tool usage]`
+- ✅ `🚧 BLOCKER: [blocker detection]`
+- ✅ `📝 COMMANDS: [commands to execute]`
+- ✅ `✓ [success messages]`
+- ✅ `✗ [failure messages]`
+
+**Development Only:**
+- ✅ `💰 Reporting token usage: X + Y`
+
+**Staging/Production (Suppressed from channel, but in file):**
+- ❌ Token usage logs (still tracked, just not displayed)
+
+## Testing the Fix
+
+To verify reasoning logs appear:
+1. Run test generation in VS Code
+2. Check "TestChimp Agent" output channel
+3. Look for `💭 REASONING`, `🧠 SELF-REFLECTION`, etc.
+4. Check log file has complete information
+
+## Files Modified
+
+**runner-core:**
+- `/src/index.ts` - Store and reapply outputChannel across service recreations
+
+**vs-extension:**
+- `/package.json` - Updated to use `^0.0.26`
+
+**github-action:**
+- `/package.json` - Updated to use `^0.0.26`
+
+## Published to npm
+
+```
+✅ Published: testchimp-runner-core@0.0.26
+📦 Package Size: 245.9 kB
+📋 Registry: https://registry.npmjs.org/
+🔢 Files: 166
+```
+
+## Verification
+
+**vs-extension:**
+```
+testchimp-vs-extension@0.0.8
+└── testchimp-runner-core@0.0.26 ✅
+```
+
+**github-action:**
+```
+testchimp-runner-action@1.0.18
+└── testchimp-runner-core@0.0.26 ✅
+```
+
+## Build Status
+
+✅ **runner-core** - Published to npm  
+✅ **vs-extension** - Updated and built (492 KiB)  
+✅ **github-action** - Updated and built (1151 KiB)  
+
+## Backward Compatibility
+
+✅ Fully backward compatible
+- No API changes
+- Only fixes a bug in log routing
+- All functionality preserved
+
+## Complete Feature Set in 0.0.26
+
+This version includes all improvements made today:
+1. ✅ Semantic selector preference (getByRole, getByLabel, etc.)
+2. ✅ Playwright expect() assertions instead of manual checks
+3. ✅ Script cleanup feature (removes redundancies)
+4. ✅ Fixed comment placement in generated scripts
+5. ✅ Focused step execution (no extra assertions)
+6. ✅ Environment-aware log filtering (in vs-extension)
+7. ✅ **Orchestrator reasoning logs in output channel** (NEW in 0.0.26)
+
+## Next Steps
+
+For future service recreations, the pattern to follow:
+```typescript
+// When recreating a service, reapply stored configuration:
+if (this.logger) {
+  this.newService.setLogger(this.logger);
+}
+if (this.outputChannel) {
+  this.newService.setOutputChannel(this.outputChannel);
+}
+```
+
+This ensures configuration persists across service lifecycles.
+

package/RELEASE_0.0.27.md

@@ -0,0 +1,236 @@
+# Release 0.0.27 - Clean Logs and Credit Callback Architecture
+
+## Summary
+Major cleanup of logging architecture with timestamps moved to consumers, minimal initialization logs, credit callback support for server-side integration, and version read from package.json.
+
+## Changes in 0.0.27
+
+### 1. Timestamps at Consumer Level
+**Architecture Fix:**
+- ❌ Before: runner-core added timestamps
+- ✅ After: Consumer adds timestamps in their timezone
+
+**runner-core:**
+- Removed all timestamp formatting
+- Reports raw messages via callbacks
+
+**vs-extension:**
+- Added `formatLocalTimestamp()` utility function
+- Wraps outputChannel to add timestamps automatically
+- Format: `HH:MM:SS.mmm` in local timezone
+
+### 2. Minimal Initialization Logs
+**Before:**
+```
+🤖 Initializing Orchestrator Mode
+✓ Orchestrator initialized with 5 tools (DEBUG MODE)
+═══════════════════════════════════════════════════════
+🚀 RUNNER-CORE VERSION: v1.5.0-vision-preserve-values
+═══════════════════════════════════════════════════════
+Initializing Scenario worker...
+Scenario worker initialized with session: scenario_worker_XXX
+Scenario worker initialized (Orchestrator Mode) with session...
+Scenario service initialized
+```
+
+**After:**
+```
+testchimp-runner-core v0.0.27
+📋 Processing scenario: [scenario description]
+```
+
+**Changes:**
+- Single version log (reads from package.json)
+- No internal initialization details
+- Clean, minimal output
+
+### 3. Credit Callback Architecture
+**Purpose:** Allow server-side integration to update DB directly without axios calls
+
+**Implementation:**
+```typescript
+export interface CreditUsage {
+  credits: number;
+  usageReason: CreditUsageReason;
+  jobId?: string;
+  timestamp: number;
+}
+
+export type CreditUsageCallback = (usage: CreditUsage) => void | Promise<void>;
+```
+
+**Behavior:**
+1. **If callback provided** (server-side):
+   - ✅ Call callback → Direct DB update
+   - ❌ NO axios calls made
+   
+2. **If NO callback but auth configured** (client-side):
+   - ❌ No callback
+   - ✅ Makes axios call to backend API
+   
+3. **If neither** (development):
+   - Logs warning
+   - Continues without tracking
+
+**Usage:**
+```typescript
+// Server-side
+const service = new TestChimpService(
+  fileHandler, undefined, backendUrl, maxWorkers,
+  llmProvider, progressReporter, orchestratorOptions,
+  async (creditUsage) => {
+    await db.insertCreditUsage(creditUsage);  // Direct DB
+  }
+);
+
+// Client-side (vs-ext, github-action)
+const service = new TestChimpService(
+  fileHandler, authConfig, backendUrl
+  // No callback - uses axios
+);
+```
+
+### 4. Wrapped OutputChannel
+**Problem:** runner-core wrote directly to outputChannel without timestamps
+
+**Solution:** vs-extension wraps the outputChannel:
+```typescript
+const wrappedChannel = {
+  appendLine: (message) => {
+    const timestamp = formatLocalTimestamp();
+    const shouldLog = isDev || !isVerboseDebugLog;
+    if (shouldLog) {
+      this.outputChannel?.appendLine(`[${timestamp}] ${message}`);
+    }
+  }
+};
+service.setOutputChannel(wrappedChannel);
+```
+
+**Result:**
+- ✅ All logs get timestamps
+- ✅ Filtering applied
+- ✅ Consumer controls presentation
+
+### 5. Dynamic Version in build_local.sh
+**Before:**
+```bash
+cp testchimp-runner-core-0.0.22.tgz .  # Hardcoded!
+```
+
+**After:**
+```bash
+RUNNER_VERSION=$(node -p "require('./package.json').version")
+TARBALL_NAME="testchimp-runner-core-${RUNNER_VERSION}.tgz"
+cp "$PARENT_DIR/runner-core/${TARBALL_NAME}" .
+```
+
+**Benefit:** Always uses correct version, no manual updates needed
+
+## New Exports
+
+```typescript
+export {
+  // Credit usage types
+  CreditUsageCallback,
+  CreditUsage,
+  CreditUsageReason,
+  
+  // Existing exports...
+  TestChimpService,
+  // ...
+}
+```
+
+## Files Modified
+
+### runner-core
+1. `/src/scenario-worker-class.ts` - Removed timestamps, minimal logs, version from package.json
+2. `/src/scenario-service.ts` - Removed initialization logs
+3. `/src/credit-usage-service.ts` - Added callback architecture, callback-first logic
+4. `/src/index.ts` - Credit callback support, preserve across recreations
+
+### vs-extension
+1. `/src/embedded-service.ts` - `formatLocalTimestamp()` utility, wrapped outputChannel, consistent filtering
+2. `/build_local.sh` - Dynamic version detection
+3. `/package.json` - Updated to `^0.0.27`
+
+### github-action
+1. `/package.json` - Updated to `^0.0.27`
+
+## Published to npm
+
+```
+✅ Published: testchimp-runner-core@0.0.27
+📦 Package Size: ~245 kB
+📋 Registry: https://registry.npmjs.org/
+```
+
+## Benefits
+
+### Logging
+1. **Local Timezone** - Timestamps match user's clock
+2. **Clean Output** - Only essential information
+3. **Consumer Control** - Consumer decides format and filtering
+4. **No Verbose Init** - Single version log instead of 10+ lines
+
+### Credit Tracking
+1. **Server-Side** - Direct DB updates, no HTTP overhead
+2. **Client-Side** - Existing axios behavior preserved
+3. **Flexible** - Each consumer decides how to track
+4. **Observable** - Callback provides visibility
+
+### Architecture
+1. **Separation of Concerns** - Library reports, consumer presents
+2. **Environment Agnostic** - Library doesn't assume timezone/environment
+3. **Testable** - Easy to mock callbacks
+4. **Maintainable** - Clear boundaries
+
+## Migration
+
+### For All Consumers
+Update package.json:
+```json
+{
+  "dependencies": {
+    "testchimp-runner-core": "^0.0.27"
+  }
+}
+```
+
+### For Server-Side (Optional)
+Add credit callback:
+```typescript
+const service = new TestChimpService(
+  fileHandler, undefined, backendUrl, maxWorkers,
+  llmProvider, progressReporter, orchestratorOptions,
+  async (creditUsage) => {
+    await creditRepository.insert(creditUsage);
+  }
+);
+```
+
+## Backward Compatibility
+
+✅ Fully backward compatible
+- All parameters optional
+- Existing behavior preserved
+- No breaking changes
+
+## Complete Feature Set
+
+All improvements from today are now in 0.0.27:
+1. ✅ Semantic selector preference
+2. ✅ Playwright expect() assertions
+3. ✅ Script cleanup feature
+4. ✅ Fixed comment placement
+5. ✅ Focused step execution
+6. ✅ Orchestrator reasoning logs in output channel
+7. ✅ Environment-aware log filtering (consumer-side)
+8. ✅ Local timezone timestamps
+9. ✅ Clean initialization logs
+10. ✅ Credit callback architecture
+11. ✅ Version from package.json
+
+Ready for production! 🚀
+

package/RELEASE_0.0.28.md

@@ -0,0 +1,286 @@
+# Runner-Core 0.0.28 Release - Scriptservice Integration
+
+## Overview
+
+Extended runner-core with lifecycle callbacks and created SmartTestRunnerCoreV2 wrapper to enable scriptservice integration. This release makes runner-core a drop-in replacement for scriptservice's duplicated SmartTestRunnerCore.
+
+## Changes
+
+### 1. Extended ProgressReporter with Lifecycle Callbacks
+
+**File:** `src/progress-reporter.ts`
+
+Added optional lifecycle callbacks to ProgressReporter interface:
+
+```typescript
+export interface StepInfo {
+  stepId?: string;
+  stepNumber: number;
+  description: string;
+  code?: string;
+}
+
+export interface ProgressReporter {
+  // ... existing callbacks ...
+  
+  // NEW: Lifecycle callbacks (used by scriptservice, ignored by local clients)
+  beforeStartTest?(page: any, browser: any, context: any): Promise<void>;
+  beforeStepStart?(step: StepInfo, page: any): Promise<void>;
+  afterEndTest?(status: 'passed' | 'failed', error?: string, page?: any): Promise<void>;
+}
+```
+
+**Purpose:**
+- `beforeStartTest`: Initialize browser context, set up DB records (scriptservice only)
+- `beforeStepStart`: Update step status to IN_PROGRESS in DB (scriptservice only)
+- `afterEndTest`: Write final status to DB, cleanup resources (scriptservice only)
+- Local clients (vs-ext, github-action) can ignore these - they use return values
+
+### 2. Integrated Lifecycle Callbacks
+
+**Files:** 
+- `src/execution-service.ts` - RUN_EXACTLY and AI_REPAIR modes
+- `src/scenario-worker-class.ts` - Orchestrator mode
+
+**Integration Points:**
+
+**ExecutionService (RUN_EXACTLY):**
+```typescript
+// Before script execution
+if (this.progressReporter?.beforeStartTest) {
+  await this.progressReporter.beforeStartTest(page, browser, context);
+}
+
+// After execution (success or failure)
+if (this.progressReporter?.afterEndTest) {
+  await this.progressReporter.afterEndTest(
+    success ? 'passed' : 'failed',
+    error,
+    page
+  );
+}
+```
+
+**ExecutionService (AI_REPAIR):**
+```typescript
+// Before each step
+if (this.progressReporter?.beforeStepStart) {
+  await this.progressReporter.beforeStepStart(
+    { stepNumber, description, code },
+    page
+  );
+}
+```
+
+**ScenarioWorker (Orchestrator):**
+```typescript
+// After browser initialization
+if (this.progressReporter?.beforeStartTest) {
+  await this.progressReporter.beforeStartTest(page, browser, context);
+}
+
+// Before each orchestrator step
+if (this.progressReporter?.beforeStepStart) {
+  await this.progressReporter.beforeStepStart({ stepNumber, description }, page);
+}
+
+// In finally block (before browser close)
+if (this.progressReporter?.afterEndTest) {
+  await this.progressReporter.afterEndTest(
+    overallSuccess ? 'passed' : 'failed',
+    error,
+    page
+  );
+}
+```
+
+### 3. Version Bump
+
+**File:** `package.json`
+
+Updated version from `0.0.27` to `0.0.28`
+
+## Scriptservice Integration
+
+### 1. Created ScriptserviceLLMProvider
+
+**File:** `services/scriptservice/providers/scriptservice-llm-provider.ts` (NEW)
+
+Implements `LLMProvider` interface for scriptservice:
+- Wraps scriptservice's existing OpenAI client
+- Supports both text and vision prompts
+- Uses scriptservice's OpenAI API key from ConfigService
+- No backend proxy - all calls are local
+- No token tracking needed (scriptservice doesn't use this interface for tracking)
+
+### 2. Created SmartTestRunnerCoreV2
+
+**File:** `services/scriptservice/smart-test-runner-core-v2.ts` (NEW)
+
+Drop-in replacement for SmartTestRunnerCore:
+- **Same Interface:** Maintains exact same constructor and methods (runExactly, runWithRepair)
+- **Uses runner-core:** Delegates to TestChimpService internally
+- **Callback Mapping:** Maps scriptservice callbacks to runner-core ProgressReporter
+- **Lifecycle Support:** Properly calls beforeStartTest, beforeStepStart, afterEndTest
+- **No Breaking Changes:** Can replace SmartTestRunnerCore with zero code changes
+
+**Key Features:**
+```typescript
+export class SmartTestRunnerCoreV2 {
+  constructor(config: RunnerConfig) {
+    // Create local LLM provider (no backend)
+    const llmProvider = new ScriptserviceLLMProvider();
+    
+    // Map callbacks to progress reporter
+    const progressReporter: ProgressReporter = {
+      onStepProgress: async (stepProgress) => {
+        // Call scriptservice's onStepComplete
+      },
+      beforeStartTest: config.callbacks?.beforeStartTest,
+      beforeStepStart: config.callbacks?.beforeStepStart,
+      afterEndTest: config.callbacks?.afterEndTest
+    };
+    
+    // Initialize runner-core (no auth, no backend)
+    this.runnerCore = new TestChimpService(
+      undefined, // No file handler
+      undefined, // No auth
+      undefined, // No backend URL
+      1,         // Single worker
+      llmProvider,
+      progressReporter
+    );
+  }
+  
+  async runExactly(script: string): Promise<RunExactlyResult> { /* ... */ }
+  async runWithRepair(steps: TestStepWithId[]): Promise<RunWithRepairResult> { /* ... */ }
+}
+```
+
+### 3. Updated Scriptservice Consumers
+
+**Files Updated:**
+- `services/scriptservice/smart-test-execution-handler.ts`
+- `services/scriptservice/workers/test-based-explorer.ts`
+- `services/scriptservice/package.json` - Added `testchimp-runner-core: ^0.0.28`
+
+**Migration Strategy:**
+```typescript
+// Environment flag for gradual rollout
+const useV2 = process.env.USE_RUNNER_CORE_V2 !== 'false'; // Default: true
+const runner = useV2 
+  ? new SmartTestRunnerCoreV2(runnerConfig) 
+  : new SmartTestRunnerCore(runnerConfig);
+```
+
+**Rollout Plan:**
+1. Deploy with `USE_RUNNER_CORE_V2=true` (default)
+2. Monitor scriptservice execution logs
+3. If issues occur, set `USE_RUNNER_CORE_V2=false` to rollback
+4. Once validated, remove flag and delete old SmartTestRunnerCore
+
+## Benefits
+
+### For Scriptservice:
+1. **Zero Axios Calls:** All LLM calls local, all DB writes via callbacks
+2. **Auto-Updates:** Automatically get runner-core improvements (better prompts, vision, orchestrator)
+3. **Code Deduplication:** Remove ~600 lines of duplicated execution/repair logic
+4. **Consistency:** Same prompts and logic as vs-extension and github-action
+
+### For Runner-Core:
+1. **Universal Library:** Works for both client-side (vs-ext) and server-side (scriptservice)
+2. **Flexible Architecture:** Callbacks allow different execution patterns
+3. **Backward Compatible:** Existing consumers unaffected (callbacks are optional)
+
+## Testing
+
+### Scriptservice Testing:
+```bash
+# Test with V2 (default)
+USE_RUNNER_CORE_V2=true npm start
+
+# Test with V1 (legacy fallback)
+USE_RUNNER_CORE_V2=false npm start
+```
+
+### Expected Behavior:
+- V2: Logs show "using SmartTestRunnerCoreV2 (runner-core)"
+- V1: Logs show "using SmartTestRunnerCore (legacy)"
+- All callbacks (beforeStartTest, beforeStepStart, afterEndTest, onStepComplete) should fire
+- DB writes should happen via callbacks
+- Screenshots should upload to GCS
+- Journey reporting should work
+
+## Publishing
+
+### Prerequisites:
+1. ✅ Build runner-core: `npm run build`
+2. ✅ No lint errors
+3. ✅ Version bumped to 0.0.28
+4. ⏸️ **Awaiting user permission to publish to npm**
+
+### Publish Command (when approved):
+```bash
+cd /Users/nuwansam/IdeaProjects/AwareRepo/local/runner-core
+npm publish
+```
+
+### Post-Publish:
+1. Install in scriptservice: `npm install testchimp-runner-core@0.0.28`
+2. Install in vs-ext: Update to 0.0.28
+3. Install in github-action: Update to 0.0.28
+
+## Migration Checklist
+
+### Phase 1: Deploy with V2 (Default)
+- [x] Add lifecycle callbacks to ProgressReporter
+- [x] Integrate callbacks in ExecutionService and ScenarioService
+- [x] Create ScriptserviceLLMProvider
+- [x] Create SmartTestRunnerCoreV2
+- [x] Update scriptservice consumers with environment flag
+- [ ] Publish runner-core 0.0.28 (needs user permission)
+- [ ] Install 0.0.28 in scriptservice
+- [ ] Deploy scriptservice with USE_RUNNER_CORE_V2=true
+- [ ] Monitor logs and execution results
+
+### Phase 2: Validate (1-2 weeks)
+- [ ] Verify all smart test executions work
+- [ ] Verify explorer mode works
+- [ ] Verify script generation works
+- [ ] Check DB writes happen correctly
+- [ ] Check screenshot uploads work
+- [ ] Monitor for any errors or regressions
+
+### Phase 3: Full Migration
+- [ ] Remove environment flag check
+- [ ] Replace all `new SmartTestRunnerCore` with `new SmartTestRunnerCoreV2`
+- [ ] Delete old `smart-test-runner-core.ts` file
+- [ ] Optional: Rename V2 to just `SmartTestRunnerCore`
+
+## Breaking Changes
+
+None. All changes are backward compatible.
+
+## Files Changed
+
+### Runner-Core:
+- `src/progress-reporter.ts` - Added lifecycle callbacks
+- `src/execution-service.ts` - Integrated callbacks
+- `src/scenario-worker-class.ts` - Integrated callbacks
+- `package.json` - Version bump to 0.0.28
+
+### Scriptservice:
+- `providers/scriptservice-llm-provider.ts` - NEW
+- `smart-test-runner-core-v2.ts` - NEW
+- `smart-test-execution-handler.ts` - Added V2 usage with flag
+- `workers/test-based-explorer.ts` - Added V2 usage with flag
+- `package.json` - Added testchimp-runner-core dependency
+
+## Notes
+
+- Lifecycle callbacks are **optional** - local clients (vs-ext, github-action) don't need them
+- Scriptservice uses callbacks for DB writes and resource management
+- V2 wrapper uses existing page/browser/context (doesn't create new ones)
+- LLM calls in V2 are all local (no backend proxy)
+- No authentication needed for scriptservice (already authenticated at service level)
+