jettypod 4.1.2 → 4.1.4

package/docs/TESTING-STRATEGY-AUDIT.md

@@ -0,0 +1,698 @@
+# Testing Strategy Audit: JettyPod Skills Workflow
+
+**Date**: 2025-11-14
+**Scope**: Feature-planning, Speed-mode, Stable-mode, Production-mode, and standalone chores
+**Focus**: What tests to write when, what tests to run when, efficiency and quality assurance
+
+---
+
+## Executive Summary
+
+**Critical Findings:**
+
+1. **Wrong tests at wrong times**: BDD step definitions are created in feature-planning but can't be verified until speed-mode completes implementation
+2. **No test execution in feature-planning**: Scenarios and step definitions are written but never validated during discovery
+3. **Speed-mode runs BDD tests correctly** but only for happy path coverage
+4. **Stable-mode has comprehensive error handling focus** but weak on test strategy specifics
+5. **Production-mode lacks concrete testing guidance** - mentions validation but no test execution strategy
+6. **Standalone chore approach (ai-test-writing-requirements.md)** has superior safety patterns but isn't integrated into workflow skills
+
+**Impact**: Broken step definitions can go undetected through entire feature-planning phase, wasting developer time in speed-mode when they discover the test infrastructure is broken.
+
+---
+
+## Current State Analysis
+
+### 1. Feature-Planning Skill
+
+**What it does with tests:**
+- Creates `.feature` file with BDD scenario (happy path only) ✅
+- Creates `step_definitions/*.steps.js` with Given/When/Then implementations ✅
+- Updates database with scenario_file path ✅
+
+**What's missing:**
+- ❌ NO verification that step definitions are syntactically correct
+- ❌ NO smoke test to ensure BDD infrastructure works
+- ❌ NO validation that Given/When/Then steps match between .feature and .steps.js
+- ❌ NO test execution to catch errors early
+
+**Quote from skill** (line 109-133):
+```
+**CRITICAL:** BDD scenarios and step definitions must ALWAYS be created together.
+Never create scenarios without step definitions.
+```
+
+**The problem**: They're created together but never validated together. This is like writing code without compiling it.
+
+**Current test timing:**
+- Write: ✅ During feature-planning
+- Run: ❌ Not until speed-mode (potentially days later)
+- Feedback loop: **Broken** - errors discovered after context is lost
+
+---
+
+### 2. Speed-Mode Skill
+
+**What it does with tests:**
+- Reads BDD scenario file and step definitions ✅
+- Implements functionality to make happy path pass ✅
+- Runs BDD tests: `npm run test:bdd -- features/[feature-slug].feature` ✅
+- Iterates until tests pass ✅
+
+**Testing approach** (lines 240-264):
+```javascript
+// Run the feature's BDD tests (scenarios + step definitions)
+const { exec } = require('child_process');
+exec('npm run test:bdd -- [feature-file].feature', (err, stdout) => {
+  // Check if step definitions pass
+  // Step definitions were created during feature discovery
+  // Your implementation must make them pass
+});
+```
+
+**What's good:**
+- ✅ Clear test execution loop
+- ✅ Uses BDD tests as acceptance criteria
+- ✅ Iterates until scenarios pass
+- ✅ Tests are run before marking chore complete
+
+**What could improve:**
+- ⚠️ No timeout handling for hung tests
+- ⚠️ No guidance on unit vs integration tests
+- ⚠️ Only validates happy path scenarios
+
+**Test types written:**
+- BDD acceptance tests (inherited from feature-planning)
+- No explicit unit tests mentioned
+- No integration tests mentioned
+
+**Test types run:**
+- BDD tests: ✅ Yes, continuously
+- Unit tests: ⚠️ Not mentioned
+- Integration tests: ⚠️ Not mentioned
+
+---
+
+### 3. Stable-Mode Skill
+
+**What it does with tests:**
+- Creates additional BDD scenarios for error handling and edge cases ✅ (lines 68-98)
+- Updates step definitions for new scenarios ✅
+- Runs all BDD tests (happy path + error/edge cases) ✅ (lines 488-553)
+- Iterates with MAX_ITERATIONS safety limit ✅
+
+**Testing approach** (lines 462-579):
+```javascript
+const MAX_ITERATIONS = 10; // Prevent infinite loops
+const TEST_TIMEOUT = 60000; // 60 second timeout per test run
+
+while (!scenarioPasses && iteration < MAX_ITERATIONS) {
+  iteration++;
+  // Modify code
+  // Run tests with timeout
+  // Check if target scenario passes
+  // Verify all scenarios still pass (no regressions)
+}
+```
+
+**What's excellent:**
+- ✅ Comprehensive error handling in test execution
+- ✅ Timeout protection (60s per test run)
+- ✅ Iteration limits prevent infinite loops
+- ✅ Regression checking - ensures new scenarios don't break old ones
+- ✅ Detailed error extraction from test output
+
+**What's good:**
+- ✅ Creates error handling scenarios
+- ✅ Creates edge case scenarios
+- ✅ Runs tests continuously during implementation
+- ✅ Validates all scenarios pass before completion
+
+**What could improve:**
+- ⚠️ Still focused only on BDD tests
+- ⚠️ No mention of unit tests for error handling logic
+- ⚠️ No guidance on test organization/structure
+
+**Test types written:**
+- BDD error handling scenarios ✅
+- BDD edge case scenarios ✅
+- Step definitions for error cases ✅
+- Unit tests for error handling: ❌ Not mentioned
+
+**Test types run:**
+- BDD tests: ✅ Yes, with comprehensive error handling
+- Unit tests: ❌ Not mentioned
+
+---
+
+### 4. Production-Mode Skill
+
+**What it does with tests:**
+- Detects context (Scenario A/B/C) ✅
+- Validates/generates production scenarios from standards ✅
+- Appends scenarios to .feature file ✅
+
+**Testing approach** (lines 295-323):
+```javascript
+// Example: If chore is about TLS enforcement
+// 1. Read the standard's acceptance criteria
+// 2. Implement code that satisfies the criteria
+// 3. Add tests based on scenario steps
+
+console.log('Run: npx cucumber-js to verify scenarios pass');
+```
+
+**What's concerning:**
+- ❌ Vague testing guidance ("Add tests based on scenario steps")
+- ❌ Manual test execution suggestion instead of automated loop
+- ❌ No iteration logic like speed/stable modes
+- ❌ No verification that tests pass before completion
+- ❌ No test timeout handling
+- ❌ No error handling in test execution
+
+**What's good:**
+- ✅ Standards-driven scenarios
+
+**What's missing:**
+- ❌ NO concrete test execution strategy
+- ❌ NO automated test verification loop
+- ❌ NO guidance on security testing vs functional testing
+- ❌ NO performance test guidance
+- ❌ NO compliance validation testing
+
+**Test types written:**
+- Production scenarios (security, scale, compliance) ✅
+- Step definitions for production scenarios: ⚠️ Implied but not explicit
+
+**Test types run:**
+- ❌ Not clearly defined
+
+---
+
+### 5. Standalone Chore Approach (ai-test-writing-requirements.md)
+
+**What it covers:**
+- Test safety patterns ✅✅✅
+- File tracking for cleanup ✅✅
+- Snapshot and restore patterns ✅✅
+- Test structure with Before/After hooks ✅✅
+
+**Key principles** (lines 6-20):
+```javascript
+// 1. Track Everything You Create
+context.createdFiles = [];
+context.createdDirs = [];
+
+// 2. Snapshot and Restore Critical Files
+this.originalSnapshot = {
+  packageJson: fs.readFileSync('package.json', 'utf-8'),
+  config: fs.readFileSync('.jettypod/config.json', 'utf-8')
+};
+
+// 3. Clean Up Test Artifacts
+// Remove all test-created files/directories
+```
+
+**What's excellent:**
+- ✅✅✅ Comprehensive safety patterns
+- ✅✅✅ Protection against test corruption of production code
+- ✅✅ File tracking and cleanup
+- ✅✅ Critical file snapshots
+- ✅✅ Error handling in cleanup (try/catch)
+- ✅ Clear dangerous patterns to avoid
+
+**What's unique:**
+- Focuses on test *safety* not test *execution*
+- Protects the codebase from test pollution
+- Assumes tests are being written, provides infrastructure
+
+**The gap:**
+- ❓ This approach is NOT integrated into any skill
+- ❓ Skills don't mention Before/After hooks
+- ❓ Skills don't emphasize test safety patterns
+- ❓ No guidance on when to use this approach
+
+---
+
+## Test Timing Analysis
+
+### What Gets Written When
+
+| Phase | Test Types Created | Status |
+|-------|-------------------|--------|
+| **Feature-planning** | BDD happy path scenario ✅<br/>Step definitions for happy path ✅ | Created but not validated |
+| **Speed-mode** | (inherits BDD tests from feature-planning) | Validates happy path |
+| **Stable-mode** | BDD error scenarios ✅<br/>BDD edge case scenarios ✅<br/>Step definitions for errors/edges ✅ | Creates and validates |
+| **Production-mode** | BDD production scenarios ✅ (security, scale, compliance)<br/>Step definitions for production ⚠️ | Created but validation unclear |
+
+**Missing entirely:**
+- ❌ Unit tests (no skill mentions them)
+- ❌ Integration tests (no skill mentions them)
+- ❌ Performance tests (mentioned in production context but not executed)
+- ❌ Security tests (mentioned in production but no concrete strategy)
+
+---
+
+### What Gets Run When
+
+| Phase | Tests Executed | Frequency | Quality |
+|-------|---------------|-----------|---------|
+| **Feature-planning** | ❌ NONE | Never | N/A |
+| **Speed-mode** | BDD happy path ✅ | Continuous loop | Good |
+| **Stable-mode** | BDD happy path ✅<br/>BDD error scenarios ✅<br/>BDD edge cases ✅ | Continuous loop with timeouts | Excellent |
+| **Production-mode** | BDD production scenarios ⚠️ | Manual? Unclear | Poor |
+
+**Problems:**
+1. **Feature-planning writes tests but never runs them** → Broken step definitions discovered later
+2. **Speed-mode only runs happy path** → Error handling untested until stable
+3. **Production-mode lacks automated verification** → Manual testing bottleneck
+
+---
+
+## Efficiency Analysis
+
+### Current Workflow Efficiency
+
+**Feature-planning:**
+- ✅ Fast: Creates scenarios and step definitions quickly
+- ❌ Risky: No validation = potential time bombs
+- ❌ Rework potential: Broken step definitions discovered in speed-mode
+
+**Speed-mode:**
+- ✅ Good: Tight feedback loop with BDD tests
+- ⚠️ Could be better: Only tests happy path
+- ✅ Iterative: Keeps trying until tests pass
+
+**Stable-mode:**
+- ✅✅ Excellent: Comprehensive test execution
+- ✅✅ Excellent: Timeout and iteration limits
+- ✅✅ Excellent: Regression checking
+- ⚠️ Could be better: Only BDD tests, no unit tests
+
+**Production-mode:**
+- ❌ Poor: Vague test execution
+- ❌ Poor: No automated verification loop
+- ❌ Poor: Manual testing suggestions
+
+---
+
+## Quality Assurance Analysis
+
+### Test Coverage by Mode
+
+**Feature-planning:**
+- Functional coverage: ✅ Happy path only (appropriate for discovery)
+- Test infrastructure: ❌ Not validated
+- Feedback timing: ❌ Delayed to speed-mode
+
+**Speed-mode:**
+- Functional coverage: ✅ Happy path (appropriate)
+- Error coverage: ❌ None (deferred to stable)
+- Regression coverage: ⚠️ Limited
+
+**Stable-mode:**
+- Functional coverage: ✅✅ Complete (happy + errors + edges)
+- Error coverage: ✅✅ Comprehensive
+- Regression coverage: ✅✅ Excellent
+- Edge case coverage: ✅✅ Explicit
+- Test safety: ❌ No mention of ai-test-writing-requirements patterns
+
+**Production-mode:**
+- Security coverage: ⚠️ Scenarios exist but validation unclear
+- Performance coverage: ❌ Not mentioned
+- Compliance coverage: ⚠️ Scenarios exist but testing unclear
+- Infrastructure coverage: ❌ Not mentioned
+
+---
+
+## Recommendations
+
+### 1. Add Test Validation to Feature-Planning (HIGH PRIORITY)
+
+**Problem**: Step definitions are created but never validated until speed-mode.
+
+**Solution**: Add smoke test execution at end of feature-planning:
+
+```javascript
+// After creating .feature and .steps.js files
+console.log('🧪 Validating BDD infrastructure...');
+
+try {
+  // Run cucumber with dry-run to validate syntax
+  const { execSync } = require('child_process');
+  execSync('npx cucumber-js --dry-run features/[feature-slug].feature', {
+    stdio: 'pipe',
+    timeout: 5000
+  });
+
+  console.log('✅ BDD infrastructure validated');
+  console.log('✅ Step definitions match scenario steps');
+} catch (err) {
+  console.error('❌ BDD infrastructure broken:');
+  console.error(err.message);
+  console.log('\nFix step definitions before proceeding to implementation.');
+  return;
+}
+```
+
+**Benefits:**
+- Catch syntax errors immediately
+- Verify step definitions match scenario steps
+- Ensure BDD infrastructure works before implementation
+- Fast feedback loop (dry-run is fast)
+
+**Cost:** ~5 seconds per feature
+
+---
+
+### 2. Integrate ai-test-writing-requirements.md Patterns (MEDIUM PRIORITY)
+
+**Problem**: Excellent test safety patterns exist but aren't used in any skill.
+
+**Solution**: Reference and require these patterns when creating step definitions:
+
+**In feature-planning** (when creating step definitions):
+```javascript
+// After writing step definitions file, add this guidance:
+console.log(`
+✅ Step definitions created: ${stepsFile}
+
+⚠️  IMPORTANT: Follow test safety patterns when implementing:
+- Track all created files in this.createdFiles
+- Track all created directories in this.createdDirs
+- Snapshot critical files before modifications
+- Use Before/After hooks for setup/cleanup
+- See: docs/ai-test-writing-requirements.md
+`);
+```
+
+**In speed-mode and stable-mode** (when implementing):
+```javascript
+// Before starting implementation:
+console.log(`
+📋 Test Safety Checklist:
+- [ ] All created files tracked for cleanup
+- [ ] All created dirs tracked for cleanup
+- [ ] Critical files (package.json, config) snapshotted
+- [ ] Before/After hooks handle setup/teardown
+- [ ] Try/catch around cleanup operations
+- [ ] No production directories in cleanup paths
+
+See: docs/ai-test-writing-requirements.md for patterns
+`);
+```
+
+**Benefits:**
+- Prevents test corruption of production code
+- Standardizes test safety across workflow
+- Reduces "test broke the codebase" incidents
+
+---
+
+### 3. Add Unit Test Guidance to Stable-Mode (MEDIUM PRIORITY)
+
+**Problem**: Stable-mode focuses on BDD tests but error handling logic often needs unit tests.
+
+**Solution**: Distinguish between BDD integration tests and unit tests for error handling:
+
+```javascript
+// After adding error handling to code
+console.log(`
+📋 Testing Strategy for Error Handling:
+
+1. BDD Tests (integration level):
+   ✅ Created error handling scenarios
+   ✅ Testing end-to-end error behavior
+
+2. Unit Tests (recommended for complex error handling):
+   - Test error detection logic in isolation
+   - Test error message formatting
+   - Test error recovery mechanisms
+   - Test error logging behavior
+
+Example: If you added error handling to validateInput():
+  - BDD test: "When I submit invalid input Then I see error message"
+  - Unit test: validateInput.test.js with edge cases
+
+Create unit tests? (y/n)
+`);
+```
+
+**Benefits:**
+- Better test granularity
+- Faster test execution (unit tests are quick)
+- Better error handling coverage
+- Easier debugging of failures
+
+---
+
+### 4. Add Automated Test Loop to Production-Mode (HIGH PRIORITY)
+
+**Problem**: Production-mode suggests manual test execution instead of automated verification.
+
+**Solution**: Add the same test execution loop as stable-mode:
+
+```javascript
+// In production-mode Step 3
+const MAX_ITERATIONS = 10;
+const TEST_TIMEOUT = 120000; // 2 minutes for production tests
+
+let iteration = 0;
+let scenarioPasses = false;
+
+while (!scenarioPasses && iteration < MAX_ITERATIONS) {
+  iteration++;
+  console.log(`\n🔄 Production Hardening Iteration ${iteration}/${MAX_ITERATIONS}`);
+
+  // 1. Implement production hardening
+  console.log('✍️  Adding production hardening...');
+  // ... use Edit tool ...
+
+  // 2. Run production scenarios with timeout
+  console.log('🧪 Running production scenarios...');
+
+  try {
+    const { stdout, stderr } = await execPromise(
+      'npx cucumber-js --tags "@production" features/[feature-slug].feature',
+      { timeout: TEST_TIMEOUT, killSignal: 'SIGTERM' }
+    );
+
+    // Check if production scenarios pass
+    const productionScenarioPassed = stdout.includes('[scenario-title]') &&
+                                      stdout.includes('✓');
+
+    if (productionScenarioPassed) {
+      console.log('✅ Production scenario passes!');
+      scenarioPasses = true;
+    } else {
+      console.log('❌ Production scenario still failing');
+      // Extract failure and iterate
+    }
+  } catch (testErr) {
+    // Handle timeout and errors
+  }
+}
+```
+
+**Benefits:**
+- Consistent testing approach across all modes
+- Automated verification before completion
+- Prevents "forgot to test" scenarios
+- Provides same quality guarantees as stable-mode
+
+---
+
+### 5. Add Test Type Guidance to All Skills (LOW PRIORITY)
+
+**Problem**: Skills don't distinguish between test types or guidance on when to use what.
+
+**Solution**: Add test type decision tree to each skill:
+
+```
+📋 Test Strategy Decision Tree:
+
+When to write BDD tests:
+✅ Testing complete user workflows
+✅ Testing integration between components
+✅ Acceptance criteria from scenarios
+✅ End-to-end behavior validation
+
+When to write Unit tests:
+✅ Testing isolated functions/methods
+✅ Testing error handling logic
+✅ Testing utility functions
+✅ Testing complex algorithms
+✅ Fast feedback needed
+
+When to write Integration tests:
+✅ Testing component interactions
+✅ Testing database operations
+✅ Testing API integrations
+✅ Testing file system operations
+
+Current skill focus: [BDD tests]
+Recommended additions: [Unit tests for error handling logic]
+```
+
+**Benefits:**
+- Clearer testing strategy
+- Better test organization
+- Appropriate test granularity
+- Faster test execution (more unit tests = faster CI)
+
+---
+
+### 6. Add Performance Testing Guidance to Production-Mode (LOW PRIORITY)
+
+**Problem**: Production-mode mentions performance but provides no testing strategy.
+
+**Solution**: Add performance testing section:
+
+```javascript
+// In production-mode after functional tests pass
+console.log(`
+🚀 Performance Validation:
+
+Production standards require:
+${standards.performance.map(s => `  - ${s.requirement}`).join('\n')}
+
+Recommended performance tests:
+1. Load testing: Can system handle expected user volume?
+2. Stress testing: Where does system break?
+3. Endurance testing: Does system degrade over time?
+4. Spike testing: How does system handle traffic spikes?
+
+Tools to consider:
+- Apache Bench (ab) for HTTP endpoints
+- Jest with performance timers for function benchmarks
+- Artillery for load testing
+- k6 for modern load testing
+
+Create performance tests? (y/n)
+`);
+```
+
+**Benefits:**
+- Explicit performance testing strategy
+- Catches performance regressions
+- Validates production readiness
+- Documents performance expectations
+
+---
+
+## Recommended Implementation Priority
+
+### Phase 1: Quick Wins (Highest ROI)
+1. ✅ **Add dry-run validation to feature-planning**
+   - Cost: 30 minutes to implement
+   - Benefit: Catch broken step definitions immediately
+   - ROI: Massive - prevents wasted time in speed-mode
+
+2. ✅ **Add automated test loop to production-mode**
+   - Cost: 1 hour to implement
+   - Benefit: Consistent quality across all modes
+   - ROI: High - prevents manual testing bottleneck
+
+### Phase 2: Quality Improvements (High Value)
+3. ✅ **Integrate ai-test-writing-requirements.md patterns**
+   - Cost: 2 hours to update all skills
+   - Benefit: Prevent test corruption of codebase
+   - ROI: High - prevents "tests broke production" incidents
+
+4. ✅ **Add unit test guidance to stable-mode**
+   - Cost: 1 hour to implement
+   - Benefit: Better test granularity and coverage
+   - ROI: Medium - improves test quality
+
+### Phase 3: Strategic Enhancements (Long-term)
+5. ⚠️ **Add test type guidance to all skills**
+   - Cost: 3 hours to implement
+   - Benefit: Better test organization
+   - ROI: Medium - improves test architecture
+
+6. ⚠️ **Add performance testing to production-mode**
+   - Cost: 4 hours to implement
+   - Benefit: Performance regression detection
+   - ROI: Medium - depends on performance requirements
+
+---
+
+## Comparative Analysis: Current vs ai-test-writing-requirements.md
+
+### Current Skills Approach
+**Focus**: Test execution and functional coverage
+**Strengths**:
+- Good BDD test execution loops (speed/stable)
+- Clear acceptance criteria
+- Iterative validation
+
+**Weaknesses**:
+- No test safety patterns
+- No file tracking/cleanup guidance
+- No snapshot/restore patterns
+- Risk of test corruption
+
+### ai-test-writing-requirements.md Approach
+**Focus**: Test safety and infrastructure
+**Strengths**:
+- Comprehensive safety patterns
+- File tracking and cleanup
+- Snapshot and restore
+- Protection against corruption
+
+**Weaknesses**:
+- Doesn't address test execution timing
+- Doesn't address test type selection
+- Not integrated into workflow
+
+### Ideal Hybrid Approach
+**Combine both**:
+1. Test safety patterns from ai-test-writing-requirements.md
+2. Test execution loops from speed/stable-mode
+3. Validation timing from feature-planning (new)
+4. Test type guidance (new)
+
+**Result**: Safe, comprehensive, well-timed testing strategy
+
+---
+
+## Summary: Testing at the Right Time
+
+### Current State Problems
+1. **Feature-planning**: Writes tests but doesn't validate them
+2. **Speed-mode**: Only tests happy path
+3. **Stable-mode**: Great test execution but missing test safety
+4. **Production-mode**: Weak automated testing
+
+### Recommended State
+1. **Feature-planning**: Write tests + dry-run validation
+2. **Speed-mode**: Test happy path + test safety patterns
+3. **Stable-mode**: Comprehensive testing + unit tests for error handling
+4. **Production-mode**: Automated test loop + performance testing
+
+### Key Insight from ai-test-writing-requirements.md
+
+The standalone chore approach reveals that JettyPod has excellent test safety patterns but they're siloed in a separate document. **The skills need to integrate these patterns** to prevent test pollution while maintaining their good test execution strategies.
+
+The ideal workflow:
+1. Write tests (feature-planning) + validate syntax
+2. Implement with safety (speed-mode) + test happy path
+3. Add robustness (stable-mode) + test errors/edges with safety patterns
+4. Harden for production (production-mode) + automated validation loop
+
+---
+
+## Conclusion
+
+**Biggest Gap**: Feature-planning creates tests but doesn't validate them.
+
+**Biggest Win**: Stable-mode has excellent test execution with timeouts and iteration limits.
+
+**Biggest Opportunity**: Integrate ai-test-writing-requirements.md safety patterns into all skills.
+
+**Recommended Next Steps**:
+1. Add dry-run validation to feature-planning (30 min, massive ROI)
+2. Add automated test loop to production-mode (1 hour, high ROI)
+3. Reference ai-test-writing-requirements.md in all skills (2 hours, prevents corruption)
+
+This would create a testing strategy that writes the right tests at the right time, runs them efficiently, and ensures both functional quality and codebase safety.

package/hooks/post-checkout

@@ -0,0 +1,17 @@
+#!/usr/bin/env node
+
+const { importAll } = require('../lib/db-import');
+
+(async () => {
+  try {
+    // Import JSON snapshots into databases after checkout
+    await importAll();
+
+    // Exit successfully - checkout should not be blocked
+    process.exit(0);
+  } catch (err) {
+    // Log error but don't block checkout
+    console.error('Post-checkout hook warning:', err.message);
+    process.exit(0);
+  }
+})();