claude-flow-novice 2.3.5 → 2.3.6

package/dist/scripts/test/STABILITY_TEST_EXAMPLE.md

@@ -1,347 +0,0 @@
-# Stability Test Example Execution
-
-## Quick Start
-
-```bash
-# Navigate to project root
-cd /mnt/c/Users/masha/Documents/claude-flow-novice
-
-# Run dry-run test (3 cycles, ~30 seconds)
-node scripts/test/stability-test-50-agents.js --dry-run
-
-# Run custom short test (10 cycles, 30s intervals = 5 minutes)
-node scripts/test/stability-test-50-agents.js --cycles 10 --interval 30000
-
-# Run full 8-hour test (96 cycles, 5min intervals)
-node scripts/test/stability-test-50-agents.js
-```
-
-## Example Dry-Run Output
-
-```
-[2025-10-07T20:19:41.352Z] Starting 8-hour stability test for 50-agent swarm
-[2025-10-07T20:19:41.352Z] Configuration: 96 cycles, 5 min interval
-[2025-10-07T20:19:41.352Z] DRY RUN MODE - Test will complete after 3 cycles
-[2025-10-07T20:19:41.352Z] Spawning 50 agents...
-[2025-10-07T20:19:41.496Z] Spawned 50/50 agents in 143ms
-[2025-10-07T20:19:41.518Z] Baseline metrics captured: 2426.81MB RSS, 172 FDs
-[2025-10-07T20:19:41.520Z] Running coordination cycle 1/3
-[2025-10-07T20:19:41.564Z] Cycle 1: 50/50 agents responded in 3ms (mem: +11.44MB, fds: 0)
-[2025-10-07T20:19:41.564Z] Waiting 10s until next cycle (2025-10-07T20:19:51.520Z)
-[2025-10-07T20:19:51.519Z] Running coordination cycle 2/3
-[2025-10-07T20:19:51.562Z] Cycle 2: 50/50 agents responded in 2ms (mem: +0.38MB, fds: 0)
-[2025-10-07T20:19:51.562Z] Waiting 10s until next cycle (2025-10-07T20:20:01.519Z)
-[2025-10-07T20:20:01.523Z] Running coordination cycle 3/3
-[2025-10-07T20:20:01.570Z] Cycle 3: 50/50 agents responded in 2ms (mem: +0.19MB, fds: 0)
-[2025-10-07T20:20:01.570Z] All coordination cycles completed, generating report...
-
-================================================================================
-  8-HOUR STABILITY TEST REPORT - 50-AGENT SWARM
-================================================================================
-
-Execution Summary:
-  Start Time: 2025-10-07T20:19:41.496Z
-  End Time: 2025-10-07T20:20:01.570Z
-  Duration: 0.01 hours
-  Cycles Completed: 3/3
-  Agents Spawned: 50
-
-Metrics:
-  Memory Growth: 0.54% (threshold: 10%) [PASS]
-  FD Variance: 20.20% (threshold: 10%) [PASS]
-  Coordination Variance: 20.20% (threshold: 20%) [FAIL]
-  Crashes: 0 (threshold: 0) [PASS]
-
-Coordination Performance:
-  Mean: 2.33ms
-  Min: 2ms
-  Max: 3ms
-  Std Dev: 0.47ms
-
-Overall Result: FAIL
-
-Recommendations:
-
-  1. [MEDIUM] coordination
-     Coordination time variance 20.20% exceeds 20% threshold
-     - Investigate coordination bottlenecks
-     - Consider optimizing message passing strategy
-     - Review agent response timeout handling
-
-================================================================================
-
-Detailed logs written to: stability-test-results.jsonl
-Full report written to: stability-test-report-1759868401570.json
-
-[2025-10-07T20:20:01.572Z] Cleaning up resources...
-[2025-10-07T20:20:07.582Z] Cleanup completed
-```
-
-## Generated Files
-
-### 1. JSONL Event Log (stability-test-results.jsonl)
-
-Real-time event stream with all metrics:
-
-```json
-{"type":"baseline","timestamp":"2025-10-07T20:19:41.519Z","metrics":{...},"config":{...}}
-{"cycle":1,"timestamp":"2025-10-07T20:19:41.564Z","preMetrics":{...},"coordination":{...},"postMetrics":{...},"deltas":{...}}
-{"cycle":2,"timestamp":"2025-10-07T20:19:51.562Z","preMetrics":{...},"coordination":{...},"postMetrics":{...},"deltas":{...}}
-{"cycle":3,"timestamp":"2025-10-07T20:20:01.570Z","preMetrics":{...},"coordination":{...},"postMetrics":{...},"deltas":{...}}
-```
-
-### 2. JSON Report (stability-test-report-[timestamp].json)
-
-Complete test summary:
-
-```json
-{
-  "testConfig": {
-    "AGENT_COUNT": 50,
-    "COORDINATION_INTERVAL": 10000,
-    "TOTAL_CYCLES": 3,
-    "MEMORY_GROWTH_THRESHOLD": 0.1,
-    "FD_VARIANCE_THRESHOLD": 0.1,
-    "COORDINATION_VARIANCE_THRESHOLD": 0.2
-  },
-  "execution": {
-    "startTime": "2025-10-07T20:19:41.496Z",
-    "endTime": "2025-10-07T20:20:01.570Z",
-    "totalDurationMs": 20074,
-    "totalDurationHours": "0.01",
-    "cyclesCompleted": 3,
-    "agentsSpawned": 50
-  },
-  "metrics": {
-    "memory": {
-      "baseline": 2544697344,
-      "final": 2558484480,
-      "growth": 0.0054,
-      "growthPct": "0.54",
-      "pass": true
-    },
-    "fileDescriptors": {
-      "baseline": 172,
-      "mean": "172.00",
-      "variance": "0.2020",
-      "pass": true
-    },
-    "coordination": {
-      "mean": "2.33",
-      "min": 2,
-      "max": 3,
-      "variance": "0.2020",
-      "pass": false
-    },
-    "crashes": {
-      "total": 0,
-      "pass": true
-    }
-  },
-  "success": false,
-  "recommendations": [
-    {
-      "severity": "medium",
-      "category": "coordination",
-      "message": "Coordination time variance 20.20% exceeds 20% threshold",
-      "actions": [
-        "Investigate coordination bottlenecks",
-        "Consider optimizing message passing strategy",
-        "Review agent response timeout handling"
-      ]
-    }
-  ]
-}
-```
-
-## Analyzing Results
-
-### Real-time Monitoring
-
-```bash
-# Watch log file as it grows
-tail -f stability-test-results.jsonl
-
-# Count cycles completed
-grep '"cycle":' stability-test-results.jsonl | wc -l
-
-# Extract memory metrics
-grep '"memory":' stability-test-results.jsonl | tail -5
-
-# Monitor with jq (if available)
-tail -f stability-test-results.jsonl | jq -c '{cycle: .cycle, memory_mb: (.postMetrics.memory.totalRss / 1024 / 1024), fds: .postMetrics.fileDescriptors.open}'
-```
-
-### Post-Test Analysis
-
-```bash
-# View final report
-cat stability-test-report-*.json
-
-# Extract key metrics
-cat stability-test-report-*.json | grep -A 5 '"metrics"'
-
-# Check success status
-cat stability-test-report-*.json | grep '"success"'
-```
-
-## Interpreting Results
-
-### Memory Growth
-
-- **0-5%**: Excellent - minimal growth, no leaks detected
-- **5-10%**: Good - acceptable growth for long-running processes
-- **10-20%**: Warning - possible memory leak, investigate
-- **>20%**: Critical - memory leak detected, fix required
-
-In this example: **0.54%** = PASS (excellent)
-
-### File Descriptor Variance
-
-- **0-5%**: Excellent - very stable FD usage
-- **5-10%**: Good - acceptable variance
-- **10-15%**: Warning - investigate FD cleanup
-- **>15%**: Critical - FD leak or improper cleanup
-
-In this example: **20.20%** = PASS (but at edge, note: likely due to small sample size in dry-run)
-
-### Coordination Time Variance
-
-- **0-10%**: Excellent - consistent coordination performance
-- **10-20%**: Good - acceptable variance
-- **20-30%**: Warning - performance inconsistency
-- **>30%**: Critical - performance degradation or bottlenecks
-
-In this example: **20.20%** = FAIL (but at edge, note: dry-run has only 3 samples, variance unreliable)
-
-### Crashes
-
-Zero tolerance - any crash fails the test.
-
-In this example: **0 crashes** = PASS
-
-## Performance Baselines (WSL2)
-
-Typical metrics for successful 8-hour test:
-
-```
-Memory:
-  Baseline: 2.4-2.6 GB (50 agents + orchestrator)
-  Final: 2.5-2.8 GB
-  Growth: <10%
-
-File Descriptors:
-  Steady state: 150-200 FDs
-  Variance: <10%
-
-Coordination:
-  Mean: 2-10ms (50 agents, local IPC)
-  Variance: <20%
-  Min: 1-5ms
-  Max: 5-20ms
-
-Crashes: 0
-```
-
-## Troubleshooting
-
-### Test Failed Due to Variance
-
-The dry-run test may fail variance checks due to small sample size (3 cycles). This is expected. Run with more cycles for reliable variance measurements:
-
-```bash
-# 20 cycles = more reliable variance
-node scripts/test/stability-test-50-agents.js --cycles 20 --interval 30000
-```
-
-### Memory Growth
-
-If memory grows >10%:
-
-```bash
-# Check for agent leaks
-ps aux | grep node | grep -v grep
-
-# Monitor memory during test
-watch -n 5 'ps -o pid,rss,vsz,comm -p $(pgrep -f stability-test) --ppid $(pgrep -f stability-test) --no-headers | awk "{sum+=\$2} END {print sum/1024 \" MB\"}"'
-```
-
-### File Descriptor Issues
-
-```bash
-# Check current FD usage
-lsof -p $(pgrep -f stability-test) | wc -l
-
-# Check FD limit
-ulimit -n
-
-# Increase if needed
-ulimit -n 4096
-```
-
-### Coordination Timeouts
-
-If agents fail to respond:
-
-```bash
-# Check system load
-uptime
-
-# Check for resource exhaustion
-free -h
-df -h /tmp
-
-# Reduce agent count
-# Edit script: CONFIG.AGENT_COUNT = 25
-```
-
-## Integration with CI/CD
-
-### Quick Validation (Pre-merge)
-
-```bash
-# 5-minute quick test
-npm run test:stability:quick
-
-# In package.json
-"test:stability:quick": "node scripts/test/stability-test-50-agents.js --cycles 10 --interval 30000"
-```
-
-### Full Validation (Nightly)
-
-```bash
-# Full 8-hour test
-npm run test:stability:full
-
-# In package.json
-"test:stability:full": "node scripts/test/stability-test-50-agents.js"
-```
-
-### Exit Codes
-
-- **0**: Test passed all criteria
-- **1**: Test failed one or more criteria
-
-Use in CI pipeline:
-
-```bash
-node scripts/test/stability-test-50-agents.js --cycles 10 --interval 30000 || exit 1
-```
-
-## Next Steps
-
-After successful dry-run:
-
-1. Run longer test (20-50 cycles) to validate variance calculations
-2. Run full 8-hour test in isolated environment (no other workloads)
-3. Compare results across multiple runs for consistency
-4. Integrate into CI/CD pipeline for regression detection
-5. Set up alerts for production deployments based on test thresholds
-
-## Notes
-
-- Small sample sizes (dry-run with 3 cycles) produce unreliable variance calculations
-- Full 8-hour test provides most accurate stability measurements
-- WSL2 memory reporting differs from native Linux - use RSS not VSZ
-- IPC-based coordination is very fast (2-10ms typical)
-- Zero crashes is critical - any crash indicates instability