claude-flow-novice 2.14.6 → 2.14.7

package/claude-assets/skills/cfn-test-runner/SKILL.md

@@ -0,0 +1,288 @@
+# CFN Test Runner Skill
+
+**Version:** 1.0.0
+**Status:** Production
+**Purpose:** Unified test execution with benchmarking and regression detection
+
+---
+
+## Overview
+
+Provides comprehensive test execution across all CFN test suites:
+- **Hello World Tests** (Layer 0-7)
+- **CFN Loop E2E Tests** (9 integration tests)
+- **Benchmark Tracking** (SQLite storage)
+- **Regression Detection** (automated alerts)
+
+**Key Features:**
+- Single command test execution
+- Historical benchmark comparison
+- Performance regression alerts
+- Git-aware baseline tracking
+- Parallel test execution
+- Comprehensive reporting
+
+---
+
+## Architecture
+
+### Skill Components
+
+```
+.claude/skills/cfn-test-runner/
+├── SKILL.md                    # This file
+├── run-all-tests.sh            # Main test runner
+├── store-benchmarks.sh         # SQLite benchmark storage
+├── detect-regressions.sh       # Regression analysis
+├── init-benchmark-db.sh        # Database initialization
+└── generate-report.sh          # Test report generation
+```
+
+---
+
+## Usage
+
+### Run All Tests
+
+```bash
+./.claude/skills/cfn-test-runner/run-all-tests.sh \
+  --suite all \
+  --benchmark \
+  --detect-regressions
+```
+
+### Run Specific Suite
+
+```bash
+# Hello World only
+./.claude/skills/cfn-test-runner/run-all-tests.sh --suite hello-world
+
+# CFN E2E only
+./.claude/skills/cfn-test-runner/run-all-tests.sh --suite cfn-e2e
+
+# Both
+./.claude/skills/cfn-test-runner/run-all-tests.sh --suite all
+```
+
+### With Regression Detection
+
+```bash
+./.claude/skills/cfn-test-runner/run-all-tests.sh \
+  --suite all \
+  --benchmark \
+  --detect-regressions \
+  --threshold 0.10
+```
+
+---
+
+## Parameters
+
+### run-all-tests.sh
+
+| Parameter | Required | Description | Default |
+|-----------|----------|-------------|---------|
+| `--suite` | No | Test suite: `all`, `hello-world`, `cfn-e2e` | `all` |
+| `--benchmark` | No | Store results in SQLite | `false` |
+| `--detect-regressions` | No | Run regression analysis | `false` |
+| `--threshold` | No | Regression threshold (%) | `0.10` |
+| `--parallel` | No | Run tests in parallel | `false` |
+| `--output` | No | Output format: `text`, `json`, `html` | `text` |
+
+---
+
+## Test Suites
+
+### Hello World (Layer 0-7)
+
+**Purpose:** Validate agent spawning patterns and coordination
+
+| Layer | Test | Duration | Critical |
+|-------|------|----------|----------|
+| 0 | Tool Validation | 60s | ✅ |
+| 1-4 | [Future] | - | - |
+| 5 | Coordinator Spawning | 120s | ✅ |
+| 6 | Review Handoff | 180s | ✅ |
+| 7 | Error Retry | 150s | ✅ |
+
+**Location:** `tests/hello-world/`
+
+### CFN Loop E2E (9 Tests)
+
+**Purpose:** Validate full CFN Loop workflow
+
+| Test | Component | Duration | Critical |
+|------|-----------|----------|----------|
+| 1 | Coordinator → Orchestrator | 30s | ✅ |
+| 2 | Loop 3 → Gate Check | 60s | ✅ |
+| 3 | Gate Pass → Loop 2 | 30s | ⚠️ |
+| 4 | Loop 2 → Product Owner | 90s | ✅ |
+| 5 | Product Owner Decision | 60s | ✅ |
+| 6 | Iteration Cycle | 120s | ✅ |
+| 7 | Redis Key Structure | 10s | ✅ |
+| 8 | Error Recovery | 60s | ⚠️ |
+| 9 | Cleanup | 10s | ⚠️ |
+
+**Location:** `tests/cfn-v3/test-e2e-cfn-loop.sh`
+
+---
+
+## Benchmark Storage
+
+### SQLite Schema
+
+```sql
+test_suites          -- Suite definitions
+test_runs            -- Execution records
+test_cases           -- Individual tests
+test_results         -- Per-test outcomes
+performance_metrics  -- Performance data
+regression_alerts    -- Detected regressions
+```
+
+**Database:** `.artifacts/test-benchmarks.db`
+
+### Stored Metrics
+
+**Test-Level:**
+- Duration (ms)
+- Status (passed/failed/skipped)
+- Assertion count
+- Error messages
+
+**Suite-Level:**
+- Total duration (seconds)
+- Success rate (%)
+- Pass/fail/skip counts
+- Git commit/branch
+
+**Performance:**
+- Agent spawn time
+- Redis operation latency
+- File I/O duration
+- Memory usage
+
+---
+
+## Regression Detection
+
+### Automatic Alerts
+
+```bash
+# Critical: Test started failing
+alert_type: test_failure
+severity: critical
+message: "TEST 5: Product Owner Decision FAILED (was passing)"
+
+# Warning: Performance degraded
+alert_type: performance_regression
+severity: warning
+message: "Loop 3 spawn time increased 25% (baseline: 5s, current: 6.25s)"
+
+# Info: Success rate dropped
+alert_type: success_rate_drop
+severity: info
+message: "Suite success rate: 88% → 77% (threshold: 10%)"
+```
+
+### Threshold Configuration
+
+```bash
+# Default: 10% regression threshold
+--threshold 0.10
+
+# Strict: 5% threshold
+--threshold 0.05
+
+# Relaxed: 20% threshold
+--threshold 0.20
+```
+
+---
+
+## Output Formats
+
+### Text (Console)
+
+```
+==========================================
+CFN Test Suite Results
+==========================================
+
+Suite: Hello World
+  Layer 0: ✅ PASSED (58.3s)
+  Layer 5: ✅ PASSED (115.7s)
+  Layer 6: ✅ PASSED (172.4s)
+  Layer 7: ✅ PASSED (148.2s)
+
+Suite: CFN E2E
+  TEST 1: ✅ PASSED (28.1s)
+  TEST 2: ✅ PASSED (54.6s)
+  TEST 3: ⚠️  SKIPPED
+  TEST 4: ✅ PASSED (87.3s)
+  TEST 5: ✅ PASSED (59.2s)
+  TEST 6: ✅ PASSED (118.9s)
+  TEST 7: ✅ PASSED (9.4s)
+
+Total: 11 tests, 9 passed, 0 failed, 2 skipped
+Duration: 851.1s
+Success Rate: 81.8%
+
+Regressions Detected: 0
+==========================================
+```
+
+### JSON (API/CI)
+
+```json
+{
+  "timestamp": "2025-11-04T02:00:00Z",
+  "git_commit": "abc123",
+  "git_branch": "main",
+  "suites": {
+    "hello-world": {
+      "total": 4,
+      "passed": 4,
+      "failed": 0,
+      "skipped": 0,
+      "duration": 494.6,
+      "success_rate": 1.0
+    },
+    "cfn-e2e": {
+      "total": 9,
+      "passed": 7,
+      "failed": 0,
+      "skipped": 2,
+      "duration": 356.5,
+      "success_rate": 0.778
+    }
+  },
+  "regressions": [],
+  "baseline_comparison": {
+    "duration_change": "+2.3%",
+    "success_rate_change": "+0.0%"
+  }
+}
+```
+
+### HTML (Report)
+
+Generates interactive HTML report with:
+- Trend graphs (Chart.js)
+- Regression highlights
+- Test history
+- Performance metrics
+
+---
+
+## Baseline Management
+
+### Establish Baseline
+
+```bash
+# First run establishes baseline
+./.claude/skills/cfn-test-runner/run-all-tests.sh --suite all --benchmark
+
+# Mark as baseline
+sqlite3 .artifacts/test-benchmarks.db << EOF
+UPDATE test_runs SET environment = 'baseline' WHERE id = (SELECT MAX(id) FROM test_runs);

package/claude-assets/skills/cfn-test-runner/detect-regressions.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# Detect test regressions
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+DB_FILE="$PROJECT_ROOT/.artifacts/test-benchmarks.db"
+
+THRESHOLD=0.10
+
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    --threshold) THRESHOLD="$2"; shift 2 ;;
+    *) shift ;;
+  esac
+done
+
+# Get latest run
+LATEST_RUN=$(sqlite3 "$DB_FILE" "SELECT id FROM test_runs ORDER BY run_timestamp DESC LIMIT 1")
+
+if [ -z "$LATEST_RUN" ]; then
+  echo "No test runs found"
+  exit 0
+fi
+
+# Get baseline (average of last 10 runs excluding latest)
+BASELINE_SUCCESS_RATE=$(sqlite3 "$DB_FILE" "
+  SELECT AVG(success_rate) FROM (
+    SELECT success_rate FROM test_runs 
+    WHERE id != $LATEST_RUN 
+    ORDER BY run_timestamp DESC 
+    LIMIT 10
+  )
+")
+
+LATEST_SUCCESS_RATE=$(sqlite3 "$DB_FILE" "SELECT success_rate FROM test_runs WHERE id = $LATEST_RUN")
+
+# Check for regression
+REGRESSION=$(awk "BEGIN {print ($BASELINE_SUCCESS_RATE - $LATEST_SUCCESS_RATE) > $THRESHOLD}")
+
+if [ "$REGRESSION" = "1" ]; then
+  DIFF=$(awk "BEGIN {printf \"%.1f\", ($BASELINE_SUCCESS_RATE - $LATEST_SUCCESS_RATE) * 100}")
+  
+  sqlite3 "$DB_FILE" << EOFSQL
+INSERT INTO regression_alerts (run_id, alert_type, severity, message)
+VALUES ($LATEST_RUN, 'success_rate_drop', 'warning', 
+        'Success rate dropped ${DIFF}% (baseline: ${BASELINE_SUCCESS_RATE}, current: ${LATEST_SUCCESS_RATE})');
+EOFSQL
+  
+  echo "⚠️  Regression detected: Success rate dropped ${DIFF}%"
+  exit 1
+fi
+
+echo "✅ No regressions detected"
+exit 0

package/claude-assets/skills/cfn-test-runner/init-benchmark-db.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Initialize benchmark database
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+DB_FILE="$PROJECT_ROOT/.artifacts/test-benchmarks.db"
+
+mkdir -p "$PROJECT_ROOT/.artifacts"
+
+sqlite3 "$DB_FILE" << 'EOFSQL'
+CREATE TABLE IF NOT EXISTS test_suites (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  name TEXT NOT NULL UNIQUE,
+  description TEXT,
+  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE IF NOT EXISTS test_runs (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  suite_id INTEGER NOT NULL,
+  run_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+  git_commit TEXT,
+  git_branch TEXT,
+  total_tests INTEGER NOT NULL,
+  passed INTEGER NOT NULL,
+  failed INTEGER NOT NULL,
+  skipped INTEGER NOT NULL,
+  duration_seconds REAL NOT NULL,
+  success_rate REAL,
+  FOREIGN KEY (suite_id) REFERENCES test_suites(id)
+);
+
+CREATE TABLE IF NOT EXISTS regression_alerts (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  run_id INTEGER NOT NULL,
+  alert_type TEXT NOT NULL,
+  severity TEXT NOT NULL,
+  message TEXT NOT NULL,
+  acknowledged INTEGER DEFAULT 0,
+  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+  FOREIGN KEY (run_id) REFERENCES test_runs(id)
+);
+
+CREATE INDEX IF NOT EXISTS idx_runs_timestamp ON test_runs(run_timestamp DESC);
+EOFSQL
+
+echo "✅ Benchmark database initialized: $DB_FILE"

package/claude-assets/skills/cfn-test-runner/run-all-tests.sh

@@ -0,0 +1,222 @@
+#!/bin/bash
+# CFN Unified Test Runner
+# Version: 1.0.0
+# Purpose: Run all test suites with benchmarking and regression detection
+
+set -euo pipefail
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+# Configuration
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+DB_FILE="$PROJECT_ROOT/.artifacts/test-benchmarks.db"
+
+# Parameters
+SUITE="all"
+BENCHMARK=false
+DETECT_REGRESSIONS=false
+THRESHOLD=0.10
+PARALLEL=false
+OUTPUT_FORMAT="text"
+
+# Parse arguments
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    --suite)
+      SUITE="$2"
+      shift 2
+      ;;
+    --benchmark)
+      BENCHMARK=true
+      shift
+      ;;
+    --detect-regressions)
+      DETECT_REGRESSIONS=true
+      shift
+      ;;
+    --threshold)
+      THRESHOLD="$2"
+      shift 2
+      ;;
+    --parallel)
+      PARALLEL=true
+      shift
+      ;;
+    --output)
+      OUTPUT_FORMAT="$2"
+      shift 2
+      ;;
+    *)
+      echo "Unknown parameter: $1"
+      exit 1
+      ;;
+  esac
+done
+
+# Initialize benchmark database if needed
+if [ "$BENCHMARK" = true ] && [ ! -f "$DB_FILE" ]; then
+  echo -e "${BLUE}[INFO]${NC} Initializing benchmark database..."
+  "$SCRIPT_DIR/init-benchmark-db.sh"
+fi
+
+# Get git info
+GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
+GIT_BRANCH=$(git branch --show-current 2>/dev/null || echo "unknown")
+
+# Start test run
+START_TIME=$(date +%s)
+TOTAL_TESTS=0
+PASSED_TESTS=0
+FAILED_TESTS=0
+SKIPPED_TESTS=0
+
+echo -e "${GREEN}=========================================="
+echo "CFN Test Suite Runner"
+echo -e "==========================================${NC}"
+echo "Suite: $SUITE"
+echo "Benchmark: $BENCHMARK"
+echo "Detect Regressions: $DETECT_REGRESSIONS"
+echo "Git: $GIT_BRANCH @ $GIT_COMMIT"
+echo ""
+
+# Run Hello World tests
+if [ "$SUITE" = "all" ] || [ "$SUITE" = "hello-world" ]; then
+  echo -e "${BLUE}[INFO]${NC} Running Hello World tests..."
+  
+  HW_START=$(date +%s)
+  HW_PASSED=0
+  HW_FAILED=0
+  HW_SKIPPED=0
+  
+  # Layer 0
+  if timeout 90 node "$PROJECT_ROOT/tests/hello-world/layer0-tool-validation.js" > /tmp/hw-layer0.log 2>&1; then
+    echo -e "${GREEN}✅ Layer 0: PASSED${NC}"
+    ((HW_PASSED++))
+  else
+    echo -e "${RED}❌ Layer 0: FAILED${NC}"
+    ((HW_FAILED++))
+  fi
+  
+  # Layer 5
+  if timeout 180 node "$PROJECT_ROOT/tests/hello-world/layer5-coordinator-spawning.js" > /tmp/hw-layer5.log 2>&1; then
+    echo -e "${GREEN}✅ Layer 5: PASSED${NC}"
+    ((HW_PASSED++))
+  else
+    echo -e "${RED}❌ Layer 5: FAILED${NC}"
+    ((HW_FAILED++))
+  fi
+  
+  # Layer 6
+  if timeout 240 node "$PROJECT_ROOT/tests/hello-world/layer6-coordinator-review.js" > /tmp/hw-layer6.log 2>&1; then
+    echo -e "${GREEN}✅ Layer 6: PASSED${NC}"
+    ((HW_PASSED++))
+  else
+    echo -e "${RED}❌ Layer 6: FAILED${NC}"
+    ((HW_FAILED++))
+  fi
+  
+  # Layer 7
+  if timeout 200 node "$PROJECT_ROOT/tests/hello-world/layer7-coordinator-error-retry.js" > /tmp/hw-layer7.log 2>&1; then
+    echo -e "${GREEN}✅ Layer 7: PASSED${NC}"
+    ((HW_PASSED++))
+  else
+    echo -e "${RED}❌ Layer 7: FAILED${NC}"
+    ((HW_FAILED++))
+  fi
+  
+  HW_END=$(date +%s)
+  HW_DURATION=$((HW_END - HW_START))
+  
+  echo -e "${BLUE}Hello World: ${HW_PASSED} passed, ${HW_FAILED} failed, ${HW_SKIPPED} skipped (${HW_DURATION}s)${NC}"
+  echo ""
+  
+  TOTAL_TESTS=$((TOTAL_TESTS + HW_PASSED + HW_FAILED + HW_SKIPPED))
+  PASSED_TESTS=$((PASSED_TESTS + HW_PASSED))
+  FAILED_TESTS=$((FAILED_TESTS + HW_FAILED))
+  SKIPPED_TESTS=$((SKIPPED_TESTS + HW_SKIPPED))
+fi
+
+# Run CFN E2E tests
+if [ "$SUITE" = "all" ] || [ "$SUITE" = "cfn-e2e" ]; then
+  echo -e "${BLUE}[INFO]${NC} Running CFN E2E tests..."
+  
+  E2E_START=$(date +%s)
+  
+  # Run E2E test and capture results
+  if timeout 600 bash "$PROJECT_ROOT/tests/cfn-v3/test-e2e-cfn-loop.sh" > /tmp/e2e-output.log 2>&1; then
+    E2E_EXIT=0
+  else
+    E2E_EXIT=$?
+  fi
+  
+  # Parse results
+  E2E_PASSED=$(grep -c "PASS" /tmp/e2e-output.log || echo 0)
+  E2E_FAILED=$(grep -c "FAIL" /tmp/e2e-output.log || echo 0)
+  E2E_SKIPPED=$(grep -c "SKIPPED" /tmp/e2e-output.log || echo 0)
+  
+  if [ $E2E_EXIT -eq 0 ]; then
+    echo -e "${GREEN}✅ CFN E2E: PASSED${NC}"
+  else
+    echo -e "${RED}❌ CFN E2E: FAILED (exit $E2E_EXIT)${NC}"
+  fi
+  
+  E2E_END=$(date +%s)
+  E2E_DURATION=$((E2E_END - E2E_START))
+  
+  echo -e "${BLUE}CFN E2E: ${E2E_PASSED} passed, ${E2E_FAILED} failed, ${E2E_SKIPPED} skipped (${E2E_DURATION}s)${NC}"
+  echo ""
+  
+  TOTAL_TESTS=$((TOTAL_TESTS + E2E_PASSED + E2E_FAILED + E2E_SKIPPED))
+  PASSED_TESTS=$((PASSED_TESTS + E2E_PASSED))
+  FAILED_TESTS=$((FAILED_TESTS + E2E_FAILED))
+  SKIPPED_TESTS=$((SKIPPED_TESTS + E2E_SKIPPED))
+fi
+
+END_TIME=$(date +%s)
+TOTAL_DURATION=$((END_TIME - START_TIME))
+SUCCESS_RATE=$(awk "BEGIN {printf \"%.1f\", ($PASSED_TESTS / $TOTAL_TESTS) * 100}")
+
+# Summary
+echo -e "${GREEN}=========================================="
+echo "Test Summary"
+echo -e "==========================================${NC}"
+echo "Total: $TOTAL_TESTS tests"
+echo "Passed: $PASSED_TESTS"
+echo "Failed: $FAILED_TESTS"
+echo "Skipped: $SKIPPED_TESTS"
+echo "Duration: ${TOTAL_DURATION}s"
+echo "Success Rate: ${SUCCESS_RATE}%"
+echo ""
+
+# Store benchmarks
+if [ "$BENCHMARK" = true ]; then
+  echo -e "${BLUE}[INFO]${NC} Storing benchmarks..."
+  "$SCRIPT_DIR/store-benchmarks.sh" \
+    --suite "$SUITE" \
+    --total "$TOTAL_TESTS" \
+    --passed "$PASSED_TESTS" \
+    --failed "$FAILED_TESTS" \
+    --skipped "$SKIPPED_TESTS" \
+    --duration "$TOTAL_DURATION" \
+    --commit "$GIT_COMMIT" \
+    --branch "$GIT_BRANCH"
+fi
+
+# Detect regressions
+if [ "$DETECT_REGRESSIONS" = true ]; then
+  echo -e "${BLUE}[INFO]${NC} Detecting regressions..."
+  if ! "$SCRIPT_DIR/detect-regressions.sh" --threshold "$THRESHOLD"; then
+    echo -e "${YELLOW}⚠️  Regressions detected!${NC}"
+  fi
+fi
+
+# Exit with failure if tests failed
+if [ $FAILED_TESTS -gt 0 ]; then
+  exit 1
+fi

package/claude-assets/skills/cfn-test-runner/store-benchmarks.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# Store test benchmarks in SQLite
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
+DB_FILE="$PROJECT_ROOT/.artifacts/test-benchmarks.db"
+
+# Parse arguments
+SUITE=""
+TOTAL=0
+PASSED=0
+FAILED=0
+SKIPPED=0
+DURATION=0
+COMMIT=""
+BRANCH=""
+
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    --suite) SUITE="$2"; shift 2 ;;
+    --total) TOTAL="$2"; shift 2 ;;
+    --passed) PASSED="$2"; shift 2 ;;
+    --failed) FAILED="$2"; shift 2 ;;
+    --skipped) SKIPPED="$2"; shift 2 ;;
+    --duration) DURATION="$2"; shift 2 ;;
+    --commit) COMMIT="$2"; shift 2 ;;
+    --branch) BRANCH="$2"; shift 2 ;;
+    *) shift ;;
+  esac
+done
+
+SUCCESS_RATE=$(awk "BEGIN {printf \"%.4f\", ($PASSED / $TOTAL)}")
+
+# Get or create suite ID
+SUITE_ID=$(sqlite3 "$DB_FILE" "SELECT id FROM test_suites WHERE name='$SUITE'")
+if [ -z "$SUITE_ID" ]; then
+  sqlite3 "$DB_FILE" "INSERT INTO test_suites (name) VALUES ('$SUITE')"
+  SUITE_ID=$(sqlite3 "$DB_FILE" "SELECT last_insert_rowid()")
+fi
+
+# Insert test run
+sqlite3 "$DB_FILE" << EOFSQL
+INSERT INTO test_runs (
+  suite_id, git_commit, git_branch, 
+  total_tests, passed, failed, skipped, 
+  duration_seconds, success_rate
+) VALUES (
+  $SUITE_ID, '$COMMIT', '$BRANCH',
+  $TOTAL, $PASSED, $FAILED, $SKIPPED,
+  $DURATION, $SUCCESS_RATE
+);
+EOFSQL
+
+echo "✅ Benchmark stored (run_id: $(sqlite3 "$DB_FILE" "SELECT last_insert_rowid()"))"