codex-genesis-harness 0.1.0 → 0.1.4

package/.codex/skills/genesis-debug-guide/checklists/flaky-test-investigation.md

@@ -0,0 +1,339 @@
+# Flaky Test Investigation Checklist
+
+**Purpose**: Systematic debugging for tests that fail intermittently. These are notoriously hard to fix but follow patterns.
+
+**Status**: MANDATORY - Complete before attempting flaky test fix.
+
+## Characteristics: Is It Actually Flaky?
+
+- [ ] **Fails inconsistently**:
+  - Run test 10 times: `for i in {1..10}; do npm test -- test.js; done`
+  - Fails ~X times out of 10
+  - Not always the same assertion fails
+  - Not 0% or 100% (if always fails, see test-failure-debug.md)
+
+- [ ] **Failure pattern exists**:
+  - Fails more often under load?
+  - Fails in CI but not locally?
+  - Fails after other tests?
+  - Fails at specific time of day?
+
+- [ ] **Previous attempts failed**:
+  - Simple retry didn't fix it
+  - Disabling parallelization didn't fix it
+  - Increasing timeout didn't fix it
+
+---
+
+## Root Cause Pattern Analysis
+
+### Pattern 1: Race Condition (Most Common)
+```javascript
+// ❌ WRONG: Race condition
+test('user updates profile', async () => {
+  updateUser({ name: 'New Name' });  // Async, not awaited!
+  expect(userName).toBe('New Name');   // Might run before update completes
+});
+
+// ✅ CORRECT: Wait for async operation
+test('user updates profile', async () => {
+  await updateUser({ name: 'New Name' });  // Wait for it
+  expect(userName).toBe('New Name');
+});
+```
+
+**Signs**:
+- [ ] Test has async operations
+- [ ] Not all async operations awaited
+- [ ] Uses `setTimeout` or `setInterval`
+- [ ] Fails under high CPU load
+- [ ] Fails more in CI (slower machines)
+
+**Fix**:
+```javascript
+// Add explicit waits
+await waitFor(() => expect(element).toBeVisible());
+
+// Use async/await properly
+const result = await asyncFunction();
+
+// Don't mix promises with callbacks
+// Use either async/await OR .then(), not both
+```
+
+### Pattern 2: Test Order Dependency
+```javascript
+// ❌ WRONG: Tests affect each other
+let globalState = 0;
+test('first test', () => { globalState = 1; });
+test('second test', () => { expect(globalState).toBe(0); });  // Fails if first runs first!
+
+// ✅ CORRECT: Each test is isolated
+beforeEach(() => { state = 0; });  // Reset before each test
+test('first test', () => { state = 1; });
+test('second test', () => { expect(state).toBe(0); });
+```
+
+**Signs**:
+- [ ] Test passes when run alone
+- [ ] Test fails when run with others
+- [ ] Test fails when run in random order
+- [ ] Global variables used
+- [ ] Database not cleaned between tests
+
+**Fix**:
+```javascript
+beforeEach(() => {
+  // Reset all state
+  jest.clearAllMocks();
+  db.clear();
+  localStorage.clear();
+});
+```
+
+### Pattern 3: Timing Issues
+```javascript
+// ❌ WRONG: Fixed timeout too short
+test('animation completes', async () => {
+  await new Promise(r => setTimeout(r, 50));  // Sometimes not enough time
+  expect(element.style.opacity).toBe(1);
+});
+
+// ✅ CORRECT: Wait for actual condition
+test('animation completes', async () => {
+  await waitFor(() => expect(element).toHaveStyle('opacity: 1'), { timeout: 5000 });
+});
+```
+
+**Signs**:
+- [ ] Uses `setTimeout` with fixed delays
+- [ ] Fails randomly
+- [ ] Fails when machine is busy
+- [ ] Fails in CI (which is slower)
+- [ ] Increasing timeout "fixes" it temporarily
+
+**Fix**:
+```javascript
+// Instead of sleep:
+await new Promise(r => setTimeout(r, 100));
+
+// Use condition-based waiting:
+await waitFor(() => expect(element).toBeVisible());
+```
+
+### Pattern 4: External Service/Network
+```javascript
+// ❌ WRONG: Calls actual API
+test('fetch user', async () => {
+  const user = await fetchRealAPI('/users/1');  // Network unreliable!
+  expect(user.name).toBe('John');
+});
+
+// ✅ CORRECT: Mock external service
+test('fetch user', async () => {
+  jest.mock('api', () => ({
+    fetch: jest.fn().mockResolvedValue({ name: 'John' })
+  }));
+  const user = await fetchUser(1);
+  expect(user.name).toBe('John');
+});
+```
+
+**Signs**:
+- [ ] Test makes network requests
+- [ ] Mock not configured properly
+- [ ] Network-dependent test
+- [ ] Fails in offline environment
+- [ ] Fails randomly (network issues)
+
+**Fix**:
+```javascript
+// Mock all external APIs
+jest.mock('node-fetch');
+nodeFetch.mockResolvedValue({ json: () => ({ ... }) });
+```
+
+### Pattern 5: Randomness/Seeding
+```javascript
+// ❌ WRONG: Random data without seed
+test('shuffle array', () => {
+  const result = shuffle([1, 2, 3]);
+  expect(result).toEqual([2, 1, 3]);  // Different order each time!
+});
+
+// ✅ CORRECT: Seed random for reproducible tests
+test('shuffle array', () => {
+  Math.seedrandom = () => 0.5;  // Set seed
+  const result = shuffle([1, 2, 3]);
+  expect(result).toEqual([3, 1, 2]);  // Same every time with same seed
+});
+```
+
+**Signs**:
+- [ ] Test uses `Math.random()`
+- [ ] Test uses `Date.now()`
+- [ ] Test uses `UUID.generate()`
+- [ ] Different assertions fail each run
+- [ ] Works locally but fails in CI
+
+**Fix**:
+```javascript
+// Seed random number generator
+beforeEach(() => {
+  jest.spyOn(Math, 'random').mockReturnValue(0.5);
+});
+
+// Mock Date.now()
+jest.spyOn(Date, 'now').mockReturnValue(1234567890);
+
+// Generate UUIDs deterministically
+jest.mock('uuid', () => ({ v4: () => 'test-uuid-123' }));
+```
+
+---
+
+## Investigation Checklist
+
+### Documentation
+- [ ] When test was added
+- [ ] When it started failing
+- [ ] Failure frequency recorded
+- [ ] Specific assertion that fails (changes?)
+
+### Reproduction
+- [ ] Test fails when run 10 times in sequence
+- [ ] Test fails when run in random order with others
+- [ ] Test fails in CI but not locally (or vice versa)
+- [ ] Test fails under high system load
+
+### Code Analysis
+- [ ] All async operations have `await`
+- [ ] No global state modifications
+- [ ] All mocks are configured
+- [ ] No `setTimeout` with fixed delays
+- [ ] No external service calls
+- [ ] No random values without seeding
+
+### Environment
+- [ ] Node version documented (local vs CI)
+- [ ] Operating system noted (different on CI?)
+- [ ] CI environment variables checked
+- [ ] Parallelization disabled for test
+- [ ] Test isolation verified
+
+### Fix Validation
+- [ ] Test passes 20 times consecutively
+- [ ] Test passes in random order with other tests
+- [ ] Test passes in CI
+- [ ] Test passes under load
+- [ ] No other tests broken
+
+---
+
+## Debugging Strategy
+
+### If Pattern 1: Race Condition
+```bash
+# Add debug logging to see timing
+npm test -- --verbose path/to/test.js
+
+# Add explicit waits
+await new Promise(r => setTimeout(r, 100));
+
+# Use waitFor with debug
+await waitFor(() => {
+  console.log('Checking condition...');
+  expect(element).toBeVisible();
+});
+```
+
+### If Pattern 2: Test Order Dependency
+```bash
+# Run tests in random order
+npm test -- --randomize path/to/test.js
+
+# Run test alone vs with others
+npm test -- test1.js test2.js test3.js  # Together
+npm test -- test2.js                     # Alone
+
+# Check beforeEach/afterEach setup
+```
+
+### If Pattern 3: Timing Issues
+```bash
+# Check system load
+# Monitor: CPU, Memory, Disk I/O during test run
+
+# Increase timeouts
+await waitFor(() => expect(...), { timeout: 10000 });
+
+# Replace sleep with condition
+// Instead of: await sleep(1000);
+await waitFor(() => expect(element).toBeVisible());
+```
+
+### If Pattern 4: External Service
+```bash
+# Verify all external calls are mocked
+grep -r "fetch\|axios\|http" src/__tests__/
+
+# Check mock setup
+console.log(jest.mock(...));
+```
+
+### If Pattern 5: Randomness
+```bash
+# Set seed for randomness
+jest.spyOn(Math, 'random').mockReturnValue(0.5);
+
+# Mock Date.now()
+jest.spyOn(Date, 'now').mockReturnValue(1234567890);
+
+# Use deterministic test data
+const testUser = { id: '123', name: 'Test' };  // Same each time
+```
+
+---
+
+## Verification: Flaky Test Fixed
+
+- [ ] Test passes 20 times consecutively
+- [ ] Test passes 20 times in random order
+- [ ] Test passes in CI environment
+- [ ] Test passes under system load
+- [ ] No debug code left
+- [ ] Updated RECOVERY_POINTS.md with pattern
+- [ ] Similar tests checked for same issue
+
+---
+
+## Recovery: Still Flaky?
+
+If still failing after fixes:
+
+1. **Add extensive logging**:
+   ```javascript
+   test('flaky test', async () => {
+     console.log('Start:', Date.now());
+     const value = getSomeValue();
+     console.log('Value:', value);
+     await waitForCondition();
+     console.log('Condition met:', Date.now());
+     expect(...).toBe(...);
+   });
+   ```
+
+2. **Disable test temporarily**:
+   ```javascript
+   test.skip('flaky test', async () => {  // Temporarily skip
+     // ...
+   });
+   ```
+
+3. **Escalate**: Mark test with TODO comment for team
+   ```javascript
+   // TODO: Fix flaky test - likely timing issue, investigate when time permits
+   test.skip('should handle concurrent updates', async () => {
+   ```
+
+4. **Create issue**: Document flaky test pattern for future fix

package/.codex/skills/genesis-debug-guide/checklists/production-bug-debug.md

@@ -0,0 +1,210 @@
+# Production Bug Debug Checklist
+
+**Purpose**: Structured investigation for production bugs with data integrity and user impact focus.
+
+**Status**: MANDATORY - Complete before `/fix-bug` for production issues.
+
+## Incident Assessment
+
+- [ ] **Bug severity determined**:
+  - Critical (service down, data loss): Requires hotfix
+  - High (significant user impact): Next release or hotfix
+  - Medium (inconvenience, workaround exists): Next release
+  - Low (cosmetic, edge case): Backlog
+
+- [ ] **User impact quantified**:
+  - How many users affected?
+  - What functionality broken?
+  - Workaround available?
+  - Revenue impact (if applicable)?
+
+- [ ] **Reproducibility confirmed**:
+  - Reproduced in staging environment first
+  - Not reproduced in production → investigation in staging
+  - Confirmed with actual user data (if possible)
+  - Environment-specific or universal?
+
+## Data Integrity Assessment
+
+- [ ] **Data corruption checked**:
+  - Database queried for anomalies
+  - No orphaned records found
+  - Foreign keys still valid
+  - Timestamps/sequences consistent
+
+- [ ] **Rollback strategy documented**:
+  - Rollback steps written down
+  - Data recovery plan if needed
+  - Backup verified
+  - Estimated rollback time
+
+- [ ] **Data migration planned** (if applicable):
+  - Affected data identified
+  - Migration strategy documented
+  - Backwards compatibility checked
+  - Dry run successful
+
+## Root Cause Investigation
+
+- [ ] **Logs collected and analyzed**:
+  - Error logs reviewed
+  - Application logs checked
+  - System logs examined
+  - Timeline of events mapped
+
+- [ ] **Recent changes identified**:
+  - Last deployment time noted
+  - Code changes reviewed (git log)
+  - Configuration changes checked
+  - Infrastructure changes noted
+
+- [ ] **Affected versions identified**:
+  - Bug introduced in version X
+  - Affects which current versions?
+  - Oldest affected version?
+  - Fixed in which version?
+
+- [ ] **Root cause identified** (or hypothesis):
+  - Specific code/config location
+  - Why was it missed in testing?
+  - Dependency issue or implementation?
+  - Environment-specific factor?
+
+## Fix Strategy
+
+- [ ] **Fix approach decided**:
+  - Minimal fix vs architectural refactor?
+  - Hotfix vs next release?
+  - Backwards compatibility needed?
+  - Breaking change implications?
+
+- [ ] **Risk assessment completed**:
+  - Could fix introduce new bugs?
+  - Dependencies to check?
+  - Side effects analyzed?
+  - Test coverage adequate?
+
+- [ ] **Verification plan documented**:
+  - Metrics to monitor after fix
+  - Rollback criteria defined
+  - Health checks configured
+  - Alert thresholds set
+
+## Testing & Validation
+
+- [ ] **Test created** (reproduces bug):
+  - Failing test demonstrates bug
+  - Test passes without bug (would fail)
+  - Test with production-like data
+  - Edge cases included
+
+- [ ] **Fix validated** (in staging):
+  - Test passes with fix
+  - All related tests pass
+  - No new test failures
+  - Performance not degraded
+
+- [ ] **Regression testing completed**:
+  - Full test suite passes
+  - Related features tested
+  - Integration scenarios verified
+  - Similar bugs searched for
+
+## Deployment Planning
+
+- [ ] **Deployment strategy determined**:
+  - Staged rollout or immediate?
+  - Feature flags needed?
+  - Canary deployment approach?
+  - Rollback testing done?
+
+- [ ] **Communication planned**:
+  - User notification (if applicable)
+  - Workaround documented
+  - ETA for fix communicated
+  - Post-fix status update planned
+
+- [ ] **Monitoring configured**:
+  - Key metrics identified
+  - Alerts set for regression
+  - Dashboard created for tracking
+  - On-call rotation aware
+
+## Release & Monitoring
+
+- [ ] **Release documentation complete**:
+  - Fix summary written
+  - User-friendly description
+  - Migration steps (if needed)
+  - Known issues documented
+
+- [ ] **Rollback plan tested**:
+  - Rollback procedure practiced
+  - Time to rollback estimated
+  - Backup verified
+  - Stakeholders notified
+
+- [ ] **Post-deployment monitoring** (first 24h):
+  - Error rate watched
+  - Performance metrics checked
+  - User reports monitored
+  - Rollback executed if issues
+
+## Prevention & Lessons
+
+- [ ] **Prevention documented**:
+  - Better test coverage needed?
+  - Contract/spec gap?
+  - Monitoring enhancement?
+  - Process improvement?
+
+- [ ] **Similar bugs searched**:
+  - Code pattern scanned
+  - Similar issues found and tracked
+  - Prevention applied broadly
+  - Team notified
+
+- [ ] **Documentation updated**:
+  - Behavior change documented
+  - Workaround removed (if applicable)
+  - System documentation updated
+  - Known issues list maintained
+
+---
+
+## Usage
+
+When debugging production bug:
+
+```bash
+# 1. Assess severity
+# - Critical: Hotfix needed
+# - High: Plan for next release + hotfix consideration
+# - Medium/Low: Standard fix process
+
+# 2. Complete this checklist
+cat .codex/skills/genesis-debug-guide/checklists/production-bug-debug.md
+
+# 3. Proceed with /fix-bug
+/fix-bug "Production issue: [description]"
+
+# 4. After fix: genesis-debug-guide auto-triggers verification
+```
+
+## Hotfix vs Regular Release Decision
+
+**Hotfix** (immediate deployment):
+- ✅ Service down or data loss risk
+- ✅ Critical user impact
+- ✅ Can be isolated fix
+- ✅ Adequate testing done
+- ✅ Rollback strategy documented
+
+**Regular Release** (next scheduled):
+- ✅ Medium/Low severity
+- ✅ Workaround available
+- ✅ Can wait for next release
+- ✅ No data integrity risk
+- ✅ More complex fix with side effects
+
+**Default**: Regular release unless severity assessment says otherwise.

package/.codex/skills/genesis-debug-guide/checklists/test-failure-debug.md

@@ -0,0 +1,158 @@
+# Test Failure Debug Checklist
+
+**Purpose**: Systematic verification before debugging test failures. Ensures test environment is clean and failure is reproducible.
+
+**Status**: MANDATORY - Complete before starting test failure investigation.
+
+## Environment Verification
+
+- [ ] **Test runs in isolation**:
+  - Run only the failing test: `npm test -- test-name.test.js`
+  - Not running entire suite (could be interference)
+  - Confirm single test failure is reproducible
+
+- [ ] **No environment contamination**:
+  - Git status clean (`git status` shows no uncommitted changes)
+  - No stale debug code or console.log statements
+  - Dependencies up to date (`npm install` recent)
+  - Node version correct (check `.nvmrc` or `package.json`)
+
+- [ ] **Test environment isolated**:
+  - No other tests running in parallel
+  - No file system state from previous tests
+  - Database fixtures clean
+  - Cache cleared if applicable
+
+## Test Fundamentals
+
+- [ ] **Test is deterministic** (not flaky):
+  - Run test 5 times consecutively
+  - Fails same way every time
+  - If fails inconsistently → switch to flaky-test-investigation.md
+
+- [ ] **Error message is clear**:
+  - Full error read (including stack trace)
+  - Assertion message understood
+  - Expected vs actual values noted
+
+- [ ] **Reproduction steps documented**:
+  - Steps to run failing test written down
+  - Any preconditions noted
+  - Environment variables documented
+
+## Test Code Analysis
+
+- [ ] **Test structure is correct**:
+  - Setup (arrange): Test data initialized
+  - Execution (act): Function called with test data
+  - Assertion (assert): Result verified
+  - Teardown: Resources cleaned up
+
+- [ ] **Mocks are properly configured**:
+  - All external dependencies mocked
+  - Mock return values match expectations
+  - Mock call counts/arguments verified
+  - Mocks reset between tests
+
+- [ ] **Test data is valid**:
+  - Test fixtures exist
+  - Data types correct (number, string, array, object)
+  - Null/undefined not unexpected
+  - Edge cases considered
+
+## Implementation Analysis
+
+- [ ] **Code under test identified**:
+  - Exact function/method being tested known
+  - File location noted
+  - Recent changes reviewed (git blame)
+
+- [ ] **Code logic reviewed**:
+  - Function parameters match test call
+  - Return type matches assertion
+  - No type mismatches
+  - Null checks in place
+
+- [ ] **Dependency chain checked**:
+  - All imported modules exist
+  - No circular dependencies
+  - Versions compatible
+  - Initialization order correct
+
+## Debug Information Gathering
+
+- [ ] **Error output analyzed**:
+  - Stack trace followed to root
+  - Line number in error matches code
+  - Message matches code (not outdated?)
+  - All context (variables, state) logged
+
+- [ ] **Variable values inspected**:
+  - Expected value printed
+  - Actual value printed
+  - Types match
+  - Data structure as expected
+
+- [ ] **Scope issues checked**:
+  - Variable in correct scope
+  - Not undefined when used
+  - Reference vs value semantics correct
+  - Closure capturing right variables
+
+## Common Issues Ruled Out
+
+- [ ] **Not a typo**:
+  - Function name spelled correctly
+  - Variable name matches
+  - No case sensitivity issues
+  - Import path correct
+
+- [ ] **Not an import issue**:
+  - Module imported correctly
+  - Correct export used (default vs named)
+  - Path relative or absolute correct
+  - No circular imports
+
+- [ ] **Not a timing issue** (for async):
+  - Promises/callbacks not mixed
+  - Async/await used correctly
+  - Test waits for async operations
+  - No race conditions
+
+## Ready to Debug
+
+- [ ] All checks above passed
+- [ ] Investigation log created (templates/debug-investigation-log.md)
+- [ ] Ready to proceed with playbook:
+  - Unit tests → playbooks/unit-test-failures.md
+  - Integration tests → playbooks/integration-test-failures.md
+  - E2E tests → playbooks/e2e-test-failures.md
+
+---
+
+## Usage
+
+Before investigating test failure:
+
+```bash
+# 1. Run test in isolation
+npm test -- NameOfTest.test.js
+
+# 2. Complete this checklist
+cat .codex/skills/genesis-debug-guide/checklists/test-failure-debug.md
+
+# 3. Select appropriate playbook
+# - Unit: playbooks/unit-test-failures.md
+# - Integration: playbooks/integration-test-failures.md
+# - E2E: playbooks/e2e-test-failures.md
+
+# 4. Create investigation log
+# cp templates/debug-investigation-log.md observability/failures/investigation-$(date +%s).md
+```
+
+## Notes
+
+- Don't skip environment verification - most failures are environmental
+- If test fails intermittently → See flaky-test-investigation.md
+- If multiple tests fail → Check test isolation first
+- If all tests fail → Check environment/dependencies