@git.zone/tstest 1.4.0 → 1.7.0

package/readme.plan.md

@@ -1,41 +1,264 @@
-# Plan for showing logs for failed tests
+# Improvement Plan for tstest and tapbundle
 
 !! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!
 
-## Goal
-When a test fails, we want to display all the console logs from that failed test in the terminal, even without the --verbose flag. This makes debugging failed tests much easier.
+## 1. Enhanced Communication Between tapbundle and tstest
 
-## Current Behavior
-- Default mode: Only shows test results, no console logs
-- Verbose mode: Shows all console logs from all tests
-- When a test fails: Only shows the error message
+### 1.1 Real-time Test Progress API
+- Create a bidirectional communication channel between tapbundle and tstest
+- Emit events for test lifecycle stages (start, progress, completion)
+- Allow tstest to subscribe to tapbundle events for better progress reporting
+- Implement a standardized message format for test metadata
 
-## Desired Behavior
-- Default mode: Shows test results, and IF a test fails, shows all console logs from that failed test
-- Verbose mode: Shows all console logs from all tests (unchanged)
-- When a test fails: Shows all console logs from that test plus the error
+### 1.2 Rich Error Reporting
+- Pass structured error objects from tapbundle to tstest
+- Include stack traces, code snippets, and contextual information
+- Support for error categorization (assertion failures, timeouts, uncaught exceptions)
+- Visual diff output for failed assertions
 
-## Implementation Plan
+## 2. Enhanced toolsArg Functionality
 
-### 1. Update TapParser
-- Store console logs for each test temporarily
-- When a test fails, mark that its logs should be shown
+### 2.1 Test Flow Control ✅
+```typescript
+tap.test('conditional test', async (toolsArg) => {
+  const result = await someOperation();
+  
+  // Skip the rest of the test
+  if (!result) {
+    return toolsArg.skip('Precondition not met');
+  }
+  
+  // Conditional skipping
+  await toolsArg.skipIf(condition, 'Reason for skipping');
+  
+  // Mark test as todo
+  await toolsArg.todo('Not implemented yet');
+});
+```
 
-### 2. Update TsTestLogger
-- Add a new method to handle failed test logs
-- Modify testConsoleOutput to buffer logs when not in verbose mode
-- When a test fails, flush the buffered logs for that test
+### 2.2 Test Metadata and Configuration ✅
+```typescript
+// Fluent syntax ✅
+tap.tags('slow', 'integration')
+  .priority('high')
+  .timeout(5000)
+  .retry(3)
+  .test('configurable test', async (toolsArg) => {
+    // Test implementation
+  });
+```
 
-### 3. Update test result handling
-- When a test fails, trigger display of all buffered logs for that test
-- Clear logs after each test completes successfully
+### 2.3 Test Data and Context Sharing ✅
+```typescript
+tap.test('data-driven test', async (toolsArg) => {
+  // Access shared context ✅
+  const sharedData = toolsArg.context.get('sharedData');
+  
+  // Set data for other tests ✅
+  toolsArg.context.set('resultData', computedValue);
+  
+  // Parameterized test data (not yet implemented)
+  const testData = toolsArg.data<TestInput>();
+  expect(processData(testData)).toEqual(expected);
+});
+```
 
-## Code Changes Needed
-1. Add log buffering to TapParser
-2. Update TsTestLogger to handle failed test logs
-3. Modify test result processing to show logs on failure
+## 3. Nested Tests and Test Suites
 
-## Files to Modify
-- `ts/tstest.classes.tap.parser.ts` - Add log buffering
-- `ts/tstest.logging.ts` - Add failed test log handling
-- `ts/tstest.classes.tap.testresult.ts` - May need to store logs
+### 3.1 Test Grouping with describe() ✅
+```typescript
+tap.describe('User Authentication', () => {
+  tap.beforeEach(async (toolsArg) => {
+    // Setup for each test in this suite
+    await toolsArg.context.set('db', await createTestDatabase());
+  });
+  
+  tap.afterEach(async (toolsArg) => {
+    // Cleanup after each test
+    await toolsArg.context.get('db').cleanup();
+  });
+  
+  tap.test('should login with valid credentials', async (toolsArg) => {
+    // Test implementation
+  });
+  
+  tap.describe('Password Reset', () => {
+    tap.test('should send reset email', async (toolsArg) => {
+      // Nested test
+    });
+  });
+});
+```
+
+### 3.2 Hierarchical Test Organization
+- Support for multiple levels of nesting
+- Inherited context and configuration from parent suites
+- Aggregated reporting for test suites
+- Suite-level lifecycle hooks
+
+## 4. Advanced Test Features
+
+### 4.1 Snapshot Testing
+```typescript
+tap.test('component render', async (toolsArg) => {
+  const output = renderComponent(props);
+  
+  // Compare with stored snapshot
+  await toolsArg.matchSnapshot(output, 'component-output');
+});
+```
+
+### 4.2 Performance Benchmarking
+```typescript
+tap.test('performance test', async (toolsArg) => {
+  const benchmark = toolsArg.benchmark();
+  
+  // Run operation
+  await expensiveOperation();
+  
+  // Assert performance constraints
+  benchmark.expect({
+    maxDuration: 1000,
+    maxMemory: '100MB'
+  });
+});
+```
+
+### 4.3 Test Fixtures and Factories ✅
+```typescript
+tap.test('with fixtures', async (toolsArg) => {
+  // Create test fixtures
+  const user = await toolsArg.fixture('user', { name: 'Test User' });
+  const post = await toolsArg.fixture('post', { author: user });
+  
+  // Use factory functions
+  const users = await toolsArg.factory('user').createMany(5);
+});
+```
+
+## 5. Test Execution Improvements
+
+### 5.1 Parallel Test Execution ✅
+- Run independent tests concurrently ✅
+- Configurable concurrency limits (via file naming convention)
+- Resource pooling for shared resources
+- Proper isolation between parallel tests ✅
+
+Implementation:
+- Tests with `para__<groupNumber>` in filename run in parallel
+- Different groups run sequentially
+- Tests without `para__` run serially
+
+### 5.2 Watch Mode
+- Automatically re-run tests on file changes
+- Intelligent test selection based on changed files
+- Fast feedback loop for development
+- Integration with IDE/editor plugins
+
+### 5.3 Advanced Test Filtering ✅ (partially)
+```typescript
+// Run tests by tags ✅
+tstest --tags "unit,fast"
+
+// Exclude tests by pattern (not yet implemented)
+tstest --exclude "**/slow/**"
+
+// Run only failed tests from last run (not yet implemented)
+tstest --failed
+
+// Run tests modified in git (not yet implemented)
+tstest --changed
+```
+
+## 6. Reporting and Analytics
+
+### 6.1 Custom Reporters
+- Plugin architecture for custom reporters
+- Built-in reporters: JSON, JUnit, HTML, Markdown
+- Real-time streaming reporters
+- Aggregated test metrics and trends
+
+### 6.2 Coverage Integration
+- Built-in code coverage collection
+- Coverage thresholds and enforcement
+- Coverage trending over time
+- Integration with CI/CD pipelines
+
+### 6.3 Test Analytics Dashboard
+- Web-based dashboard for test results
+- Historical test performance data
+- Flaky test detection
+- Test impact analysis
+
+## 7. Developer Experience
+
+### 7.1 Better Error Messages
+- Clear, actionable error messages
+- Suggestions for common issues
+- Links to documentation
+- Code examples in error output
+
+### 7.2 Interactive Mode (Needs Detailed Specification)
+- REPL for exploring test failures
+  - Need to define: How to enter interactive mode? When tests fail?
+  - What commands/features should be available in the REPL?
+- Debugging integration  
+  - Node.js inspector protocol integration?
+  - Breakpoint support?
+- Step-through test execution
+  - Pause between tests? 
+  - Step into/over/out functionality?
+- Interactive test data manipulation
+  - Modify test inputs on the fly?
+  - Inspect intermediate values?
+
+### 7.3 ~~VS Code Extension~~ (Scratched)
+- ~~Test explorer integration~~
+- ~~Inline test results~~
+- ~~CodeLens for running individual tests~~
+- ~~Debugging support~~
+
+## Implementation Phases
+
+### Phase 1: Core Enhancements (Priority: High) ✅
+1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry) ✅
+2. Add basic test grouping with describe() ✅
+3. Improve error reporting between tapbundle and tstest ✅
+
+### Phase 2: Advanced Features (Priority: Medium)
+1. Implement nested test suites ✅ (basic describe support)
+2. Add snapshot testing ✅
+3. Create test fixture system ✅
+4. Implement parallel test execution ✅
+
+### Phase 3: Developer Experience (Priority: Medium)
+1. Add watch mode
+2. Implement custom reporters
+3. ~~Create VS Code extension~~ (Scratched)
+4. Add interactive debugging (Needs detailed spec first)
+
+### Phase 4: Analytics and Performance (Priority: Low)
+1. Build test analytics dashboard
+2. Add performance benchmarking
+3. Implement coverage integration
+4. Create trend analysis tools
+
+## Technical Considerations
+
+### API Design Principles
+- Maintain backward compatibility
+- Progressive enhancement approach
+- Opt-in features to avoid breaking changes
+- Clear migration paths for new features
+
+### Performance Goals
+- Minimal overhead for test execution
+- Efficient parallel execution
+- Fast test discovery
+- Optimized browser test bundling
+
+### Integration Points
+- Clean interfaces between tstest and tapbundle
+- Extensible plugin architecture
+- Standard test result format
+- Compatible with existing CI/CD tools

package/ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@git.zone/tstest',
-  version: '1.4.0',
+  version: '1.7.0',
   description: 'a test utility to run tests that match test/**/*.ts'
 }

package/ts/index.ts

@@ -12,6 +12,7 @@ export const runCli = async () => {
   const args = process.argv.slice(2);
   const logOptions: LogOptions = {};
   let testPath: string | null = null;
+  let tags: string[] = [];
   
   // Parse options
   for (let i = 0; i < args.length; i++) {
@@ -36,6 +37,11 @@ export const runCli = async () => {
       case '--logfile':
         logOptions.logFile = true; // Set this as a flag, not a value
         break;
+      case '--tags':
+        if (i + 1 < args.length) {
+          tags = args[++i].split(',');
+        }
+        break;
       default:
         if (!arg.startsWith('-')) {
           testPath = arg;
@@ -52,6 +58,7 @@ export const runCli = async () => {
     console.error('  --no-color      Disable colored output');
     console.error('  --json          Output results as JSON');
     console.error('  --logfile       Write logs to .nogit/testlogs/[testfile].log');
+    console.error('  --tags          Run only tests with specified tags (comma-separated)');
     process.exit(1);
   }
   
@@ -66,6 +73,6 @@ export const runCli = async () => {
     executionMode = TestExecutionMode.DIRECTORY;
   }
   
-  const tsTestInstance = new TsTest(process.cwd(), testPath, executionMode, logOptions);
+  const tsTestInstance = new TsTest(process.cwd(), testPath, executionMode, logOptions, tags);
   await tsTestInstance.run();
 };

package/ts/tspublish.json

@@ -0,0 +1,3 @@
+{
+  "order": 2
+}

package/ts/tstest.classes.tap.parser.ts

@@ -16,7 +16,7 @@ export class TapParser {
   expectedTests: number;
   receivedTests: number;
 
-  testStatusRegex = /(ok|not\sok)\s([0-9]+)\s-\s(.*)\s#\stime=(.*)ms$/;
+  testStatusRegex = /(ok|not\sok)\s([0-9]+)\s-\s(.*)(\s#\s(.*))?$/;
   activeTapTestResult: TapTestResult;
   collectingErrorDetails: boolean = false;
   currentTestError: string[] = [];
@@ -78,14 +78,33 @@ export class TapParser {
         })();
 
         const testSubject = regexResult[3];
-        const testDuration = parseInt(regexResult[4]);
-
-        // test for protocol error
-        if (testId !== this.activeTapTestResult.id) {
-          if (this.logger) {
-            this.logger.error('Something is strange! Test Ids are not equal!');
+        const testMetadata = regexResult[5]; // This will be either "time=XXXms" or "SKIP reason" or "TODO reason"
+        
+        let testDuration = 0;
+        let isSkipped = false;
+        let isTodo = false;
+        
+        if (testMetadata) {
+          const timeMatch = testMetadata.match(/time=(\d+)ms/);
+          const skipMatch = testMetadata.match(/SKIP\s*(.*)/);
+          const todoMatch = testMetadata.match(/TODO\s*(.*)/);
+          
+          if (timeMatch) {
+            testDuration = parseInt(timeMatch[1]);
+          } else if (skipMatch) {
+            isSkipped = true;
+          } else if (todoMatch) {
+            isTodo = true;
           }
         }
+
+        // test for protocol error - disabled as it's not critical
+        // The test ID mismatch can occur when tests are filtered, skipped, or use todo
+        // if (testId !== this.activeTapTestResult.id) {
+        //   if (this.logger) {
+        //     this.logger.error('Something is strange! Test Ids are not equal!');
+        //   }
+        // }
         this.activeTapTestResult.setTestResult(testOk);
 
         if (testOk) {
@@ -107,27 +126,41 @@ export class TapParser {
           this.activeTapTestResult.addLogLine(logLine);
         }
         
-        // Check if we're collecting error details
-        if (this.collectingErrorDetails) {
-          // Check if this line is an error detail (starts with Error: or has stack trace characteristics)
-          if (logLine.trim().startsWith('Error:') || logLine.trim().match(/^\s*at\s/)) {
-            this.currentTestError.push(logLine);
-          } else if (this.currentTestError.length > 0) {
-            // End of error details, show the error
-            const errorMessage = this.currentTestError.join('\n');
+        // Check for snapshot communication
+        const snapshotMatch = logLine.match(/###SNAPSHOT###(.+)###SNAPSHOT###/);
+        if (snapshotMatch) {
+          const base64Data = snapshotMatch[1];
+          try {
+            const snapshotData = JSON.parse(Buffer.from(base64Data, 'base64').toString());
+            this.handleSnapshot(snapshotData);
+          } catch (error) {
             if (this.logger) {
-              this.logger.testErrorDetails(errorMessage);
+              this.logger.testConsoleOutput(`Error parsing snapshot data: ${error.message}`);
             }
-            this.collectingErrorDetails = false;
-            this.currentTestError = [];
           }
-        }
-        
-        // Don't output TAP error details as console output when we're collecting them
-        if (!this.collectingErrorDetails || (!logLine.trim().startsWith('Error:') && !logLine.trim().match(/^\s*at\s/))) {
-          if (this.logger) {
-            // This is console output from the test file, not TAP protocol
-            this.logger.testConsoleOutput(logLine);
+        } else {
+          // Check if we're collecting error details
+          if (this.collectingErrorDetails) {
+            // Check if this line is an error detail (starts with Error: or has stack trace characteristics)
+            if (logLine.trim().startsWith('Error:') || logLine.trim().match(/^\s*at\s/)) {
+              this.currentTestError.push(logLine);
+            } else if (this.currentTestError.length > 0) {
+              // End of error details, show the error
+              const errorMessage = this.currentTestError.join('\n');
+              if (this.logger) {
+                this.logger.testErrorDetails(errorMessage);
+              }
+              this.collectingErrorDetails = false;
+              this.currentTestError = [];
+            }
+          }
+          
+          // Don't output TAP error details as console output when we're collecting them
+          if (!this.collectingErrorDetails || (!logLine.trim().startsWith('Error:') && !logLine.trim().match(/^\s*at\s/))) {
+            if (this.logger) {
+              // This is console output from the test file, not TAP protocol
+              this.logger.testConsoleOutput(logLine);
+            }
           }
         }
       }
@@ -205,6 +238,59 @@ export class TapParser {
   public async handleTapLog(tapLog: string) {
     this._processLog(tapLog);
   }
+  
+  /**
+   * Handle snapshot data from the test
+   */
+  private async handleSnapshot(snapshotData: { path: string; content: string; action: string }) {
+    try {
+      const smartfile = await import('@push.rocks/smartfile');
+      
+      if (snapshotData.action === 'compare') {
+        // Try to read existing snapshot
+        try {
+          const existingSnapshot = await smartfile.fs.toStringSync(snapshotData.path);
+          if (existingSnapshot !== snapshotData.content) {
+            // Snapshot mismatch
+            if (this.logger) {
+              this.logger.testConsoleOutput(`Snapshot mismatch: ${snapshotData.path}`);
+              this.logger.testConsoleOutput(`Expected:\n${existingSnapshot}`);
+              this.logger.testConsoleOutput(`Received:\n${snapshotData.content}`);
+            }
+            // TODO: Communicate failure back to the test
+          } else {
+            if (this.logger) {
+              this.logger.testConsoleOutput(`Snapshot matched: ${snapshotData.path}`);
+            }
+          }
+        } catch (error: any) {
+          if (error.code === 'ENOENT') {
+            // Snapshot doesn't exist, create it
+            const dirPath = snapshotData.path.substring(0, snapshotData.path.lastIndexOf('/'));
+            await smartfile.fs.ensureDir(dirPath);
+            await smartfile.memory.toFs(snapshotData.content, snapshotData.path);
+            if (this.logger) {
+              this.logger.testConsoleOutput(`Snapshot created: ${snapshotData.path}`);
+            }
+          } else {
+            throw error;
+          }
+        }
+      } else if (snapshotData.action === 'update') {
+        // Update snapshot
+        const dirPath = snapshotData.path.substring(0, snapshotData.path.lastIndexOf('/'));
+        await smartfile.fs.ensureDir(dirPath);
+        await smartfile.memory.toFs(snapshotData.content, snapshotData.path);
+        if (this.logger) {
+          this.logger.testConsoleOutput(`Snapshot updated: ${snapshotData.path}`);
+        }
+      }
+    } catch (error: any) {
+      if (this.logger) {
+        this.logger.testConsoleOutput(`Error handling snapshot: ${error.message}`);
+      }
+    }
+  }
 
   public async evaluateFinalResult() {
     this.receivedTests = this.testStore.length;

package/ts/tstest.classes.testdirectory.ts

@@ -99,4 +99,43 @@ export class TestDirectory {
     }
     return testFilePaths;
   }
+
+  /**
+   * Get test files organized by parallel execution groups
+   * @returns An object with grouped tests
+   */
+  async getTestFileGroups(): Promise<{
+    serial: string[];
+    parallelGroups: { [groupName: string]: string[] };
+  }> {
+    await this._init();
+    
+    const result = {
+      serial: [] as string[],
+      parallelGroups: {} as { [groupName: string]: string[] }
+    };
+    
+    for (const testFile of this.testfileArray) {
+      const filePath = testFile.path;
+      const fileName = plugins.path.basename(filePath);
+      
+      // Check if file has parallel group pattern
+      const parallelMatch = fileName.match(/\.para__(\d+)\./);
+      
+      if (parallelMatch) {
+        const groupNumber = parallelMatch[1];
+        const groupName = `para__${groupNumber}`;
+        
+        if (!result.parallelGroups[groupName]) {
+          result.parallelGroups[groupName] = [];
+        }
+        result.parallelGroups[groupName].push(filePath);
+      } else {
+        // File runs serially
+        result.serial.push(filePath);
+      }
+    }
+    
+    return result;
+  }
 }