claude-flow-novice 2.14.3 → 2.14.4

package/.claude/skills/cfn-redis-coordination/HEARTBEAT_MONITORING.md

@@ -1,267 +0,0 @@
-# Heartbeat Monitoring for CFN Loop Orchestration
-
-## Overview
-
-The orchestrator includes built-in heartbeat monitoring to detect hung or unresponsive agents during BLPOP waiting periods. This feature provides early warning of agent failures and enables quorum-aware decision making.
-
-## Features
-
-- **Periodic Health Checks**: Monitors agent heartbeats every 30 seconds
-- **Missed Beat Tracking**: Tracks consecutive missed heartbeats per agent
-- **Quorum-Aware Decisions**: Determines if the loop can continue without hung agents
-- **Automatic Recovery**: Resets counters when agents recover
-- **Graceful Shutdown**: Monitors stop cleanly when orchestrator shuts down
-
-## Architecture
-
-### Components
-
-1. **Heartbeat Check Function** (`check_agent_heartbeat`)
-   - Checks Redis key: `swarm:{task_id}:{agent_id}:heartbeat`
-   - Returns 0 if heartbeat exists, 1 if missing
-
-2. **Loop Health Check** (`check_heartbeats_loop`)
-   - Checks all agents in a loop
-   - Increments missed heartbeat counter
-   - Warns after 2 consecutive misses (60 seconds)
-   - Evaluates quorum impact
-
-3. **Background Monitor** (`start_heartbeat_monitor`)
-   - Runs in background subprocess
-   - Checks every 30 seconds
-   - Stops via marker file removal
-   - Respects SHUTDOWN_REQUESTED flag
-
-4. **Monitor Cleanup** (`stop_heartbeat_monitor`)
-   - Removes marker file
-   - Terminates background process
-   - Called during shutdown
-
-### Heartbeat Data Format
-
-```json
-{
-  "timestamp": 1760898665,
-  "status": "working",
-  "iteration": 1,
-  "task": "implementing feature X"
-}
-```
-
-## Usage
-
-### Agent Side (Publishing Heartbeats)
-
-Agents should publish heartbeats every 20-30 seconds:
-
-```bash
-# Set heartbeat with 60s TTL
-HEARTBEAT=$(jq -n \
-  --arg ts "$(date +%s)" \
-  --arg status "working" \
-  --arg iteration "1" \
-  '{timestamp: ($ts | tonumber), status: $status, iteration: ($iteration | tonumber)}')
-
-redis-cli SET "swarm:${TASK_ID}:${AGENT_ID}:heartbeat" "$HEARTBEAT" EX 60
-```
-
-### Orchestrator Side (Monitoring)
-
-The orchestrator automatically starts/stops monitors during each loop:
-
-```bash
-# Loop 3 monitoring
-LOOP3_HEARTBEAT_MONITOR_PID=$(start_heartbeat_monitor "$TASK_ID" "loop3" "${LOOP3_AGENTS[@]}")
-
-# ... wait for agents ...
-
-stop_heartbeat_monitor "$TASK_ID" "loop3" "$LOOP3_HEARTBEAT_MONITOR_PID"
-```
-
-## Monitoring Output
-
-### Normal Operation
-
-```
-[Loop 3] Starting heartbeat monitor (checking every 30s)...
-```
-
-### Agent Appears Hung
-
-```
-[2025-10-19T18:30:00Z] [loop3] ⚠️ agent-1 appears hung (no heartbeat for 60s)
-[2025-10-19T18:30:00Z] [loop3] ℹ️ Continuing with quorum (2/2 agents)
-```
-
-### Quorum at Risk
-
-```
-[2025-10-19T18:30:00Z] [loop3] ⚠️ agent-2 appears hung (no heartbeat for 60s)
-[2025-10-19T18:30:00Z] [loop3] ⚠️ Cannot meet quorum without agent-2 (1/2)
-```
-
-## Configuration
-
-### Monitoring Interval
-
-Default: 30 seconds
-
-To change, edit the `sleep` duration in `start_heartbeat_monitor`:
-
-```bash
-sleep 30  # Check every 30s
-```
-
-### Missed Heartbeat Threshold
-
-Default: 2 consecutive misses (60 seconds)
-
-To change, edit the threshold in `check_heartbeats_loop`:
-
-```bash
-if [ ${MISSED_HEARTBEATS["$AGENT"]} -ge 2 ]; then
-```
-
-### Heartbeat TTL
-
-Default: 60 seconds
-
-Agents should set TTL when publishing:
-
-```bash
-redis-cli SET "swarm:${TASK_ID}:${AGENT_ID}:heartbeat" "$DATA" EX 60
-```
-
-## Integration Points
-
-### Cleanup Handler
-
-Monitors are automatically stopped during shutdown:
-
-```bash
-function cleanup_and_exit() {
-  # Stop heartbeat monitors if running
-  if [ -n "${LOOP3_HEARTBEAT_MONITOR_PID:-}" ]; then
-    stop_heartbeat_monitor "$TASK_ID" "loop3" "$LOOP3_HEARTBEAT_MONITOR_PID"
-  fi
-  if [ -n "${LOOP2_HEARTBEAT_MONITOR_PID:-}" ]; then
-    stop_heartbeat_monitor "$TASK_ID" "loop2" "$LOOP2_HEARTBEAT_MONITOR_PID"
-  fi
-}
-```
-
-### Global Variables
-
-```bash
-LOOP3_HEARTBEAT_MONITOR_PID=""
-LOOP2_HEARTBEAT_MONITOR_PID=""
-declare -A MISSED_HEARTBEATS
-```
-
-## Testing
-
-### Unit Tests
-
-```bash
-./.claude/skills/redis-coordination/tests/test-heartbeat-simple.sh
-```
-
-Tests:
-1. Active heartbeat detection
-2. Missing heartbeat detection
-3. Missed heartbeat counter increment
-4. Counter reset on recovery
-
-### Manual Testing
-
-```bash
-# Set up test heartbeat
-redis-cli SET "swarm:test-task:test-agent:heartbeat" '{"timestamp": 1234567890}' EX 60
-
-# Source functions
-source ./.claude/skills/redis-coordination/heartbeat-functions.sh
-
-# Check heartbeat
-check_agent_heartbeat "test-agent" "test-task"
-echo $?  # Should be 0 (success)
-```
-
-## Best Practices
-
-### For Agents
-
-1. **Publish regularly**: Every 20-30 seconds
-2. **Use appropriate TTL**: 60 seconds (2x publish interval)
-3. **Include metadata**: Status, iteration, current task
-4. **Handle errors**: Continue if Redis unavailable
-
-### For Orchestrators
-
-1. **Start monitors early**: Before waiting for agents
-2. **Stop monitors promptly**: After agents complete
-3. **Check quorum impact**: Before making decisions
-4. **Log appropriately**: Warn for hung agents, not for normal timeouts
-
-## Troubleshooting
-
-### Monitor Not Detecting Hung Agents
-
-**Symptoms**: Agents appear hung but no warnings
-
-**Causes**:
-1. Monitor not started
-2. Heartbeat check interval too long
-3. Missed heartbeat threshold too high
-
-**Solutions**:
-- Verify monitor PID is set
-- Check monitor marker file exists
-- Review threshold values
-
-### False Positives
-
-**Symptoms**: Warnings for healthy agents
-
-**Causes**:
-1. Heartbeat publish interval too long
-2. Heartbeat TTL too short
-3. Network latency issues
-
-**Solutions**:
-- Reduce heartbeat interval (e.g., 15s)
-- Increase TTL (e.g., 90s)
-- Increase missed heartbeat threshold
-
-### Monitor Not Stopping
-
-**Symptoms**: Background processes remain after completion
-
-**Causes**:
-1. Marker file not removed
-2. Process not killed
-3. Cleanup not called
-
-**Solutions**:
-- Check for marker files: `ls /tmp/heartbeat-monitor-*.active`
-- Kill processes: `pkill -f heartbeat-monitor`
-- Verify cleanup handler is registered
-
-## Performance Impact
-
-- **CPU**: Negligible (~0.01% per monitor)
-- **Network**: ~1 Redis GET per agent per 30s
-- **Memory**: ~1KB per agent for tracking state
-
-## Future Enhancements
-
-1. **Adaptive Intervals**: Reduce check frequency for stable agents
-2. **Health Scores**: Track reliability over time
-3. **Auto-Retry**: Wake hung agents with lower priority
-4. **Metrics Export**: Publish heartbeat stats to monitoring system
-5. **Dead Letter Queue Integration**: Automatic DLQ writes for consistently hung agents
-
-## Related Documentation
-
-- [Redis Coordination Skill](./SKILL.md)
-- [CFN Loop Orchestration](./orchestrate-cfn-loop.sh)
-- [Waiting Mode Documentation](../../CLAUDE.md#redis-waiting-mode-zero-token-agent-coordination)

package/.claude/skills/cfn-redis-coordination/LOGGING.md

@@ -1,260 +0,0 @@
-# CFN Loop Logging System
-
-## Overview
-
-The CFN Loop logging system provides comprehensive visibility into agent execution, decisions, and errors. All logs are stored in SQLite for efficient querying and analysis by AI agents.
-
-## Database Location
-
-```bash
-Default: claude-flow-novice/data/cfn-loop.db
-Custom:  Set DB_PATH environment variable
-```
-
-## Schema
-
-```sql
-CREATE TABLE cfn_loop_logs (
-  id INTEGER PRIMARY KEY AUTOINCREMENT,
-  task_id TEXT NOT NULL,              -- Task/swarm identifier
-  timestamp TEXT DEFAULT (datetime('now')),  -- ISO 8601 timestamp
-  event_type TEXT NOT NULL,           -- Event category
-  loop TEXT,                          -- loop3, loop2, product_owner, coordinator
-  agent_id TEXT,                      -- Agent identifier (e.g., coder-1-1)
-  iteration INTEGER,                  -- CFN loop iteration number
-  details TEXT,                       -- JSON payload with event-specific data
-  level TEXT DEFAULT 'INFO'           -- DEBUG, INFO, WARN, ERROR
-);
-
--- Indexes for fast queries
-CREATE INDEX idx_task_id ON cfn_loop_logs(task_id);
-CREATE INDEX idx_event_type ON cfn_loop_logs(event_type);
-CREATE INDEX idx_timestamp ON cfn_loop_logs(timestamp);
-CREATE INDEX idx_level ON cfn_loop_logs(level);
-```
-
-## Event Types
-
-| Event Type | Description | Level | Details Payload |
-|-----------|-------------|-------|-----------------|
-| `swarm_init` | CFN loop initialization | INFO | `{mode, loop3_agents, loop2_agents, product_owner, max_iterations, gate_threshold, consensus_threshold}` |
-| `agent_spawn` | Agent process started | INFO | `{agent_type, timeout}` |
-| `agent_complete` | Agent successfully completed | INFO | `{confidence, confidence_source, files_changed, latency_ms}` |
-| `agent_failure` | Agent execution failed | ERROR | `{error, output}` |
-| `gate_check` | Loop 3 gate validation | INFO/WARN | `{consensus, threshold, result: PASS\|FAIL, decision?}` |
-| `po_decision` | Product Owner strategic decision | INFO | `{decision: PROCEED\|ITERATE\|ABORT, reasoning, confidence}` |
-| `parameter_error` | Invalid parameters passed to tool | ERROR | `{error, command}` |
-
-## Usage
-
-### Logging Events (Orchestrator/Scripts)
-
-```bash
-# Log swarm initialization
-./.claude/skills/redis-coordination/log-event.sh \
-  --task-id "cfn-task-123" \
-  --event-type "swarm_init" \
-  --details '{"mode": "standard", "loop3_agents": "coder", "loop2_agents": "reviewer"}' \
-  --level "INFO"
-
-# Log agent spawn
-./.claude/skills/redis-coordination/log-event.sh \
-  --task-id "cfn-task-123" \
-  --event-type "agent_spawn" \
-  --loop "loop3" \
-  --agent-id "coder-1-1" \
-  --iteration 1 \
-  --details '{"agent_type": "coder", "timeout": 900}' \
-  --level "INFO"
-
-# Log error
-./.claude/skills/redis-coordination/log-event.sh \
-  --task-id "cfn-task-123" \
-  --event-type "agent_failure" \
-  --loop "loop3" \
-  --agent-id "coder-1-1" \
-  --iteration 1 \
-  --details '{"error": "timeout", "output": "Agent exceeded 900s timeout"}' \
-  --level "ERROR"
-```
-
-### Querying Logs (AI Agents/Debugging)
-
-```bash
-# Get all logs for a task
-./query-logs.sh --task-id "cfn-task-123"
-
-# Get only errors
-./query-logs.sh --task-id "cfn-task-123" --level ERROR
-
-# Get Loop 3 agent spawns
-./query-logs.sh --task-id "cfn-task-123" --event-type agent_spawn --loop loop3
-
-# Get Product Owner decisions
-./query-logs.sh --task-id "cfn-task-123" --event-type po_decision
-
-# Get latest 10 events in table format
-./query-logs.sh --task-id "cfn-task-123" --limit 10 --format table
-
-# Get events for specific iteration
-./query-logs.sh --task-id "cfn-task-123" --iteration 2
-
-# Get events for specific agent
-./query-logs.sh --task-id "cfn-task-123" --agent-id "coder-1-1"
-```
-
-### Output Formats
-
-**JSON (default):**
-```json
-[
-  {
-    "id": 1,
-    "task_id": "cfn-task-123",
-    "timestamp": "2025-10-21T10:30:00Z",
-    "event_type": "agent_spawn",
-    "loop": "loop3",
-    "agent_id": "coder-1-1",
-    "iteration": 1,
-    "details": "{\"agent_type\": \"coder\", \"timeout\": 900}",
-    "level": "INFO"
-  }
-]
-```
-
-**Table:**
-```
-id  task_id        timestamp             event_type     loop    agent_id     iteration  level
-1   cfn-task-123   2025-10-21T10:30:00Z  agent_spawn   loop3   coder-1-1    1          INFO
-2   cfn-task-123   2025-10-21T10:45:00Z  agent_complete loop3   coder-1-1    1          INFO
-```
-
-**CSV:**
-```csv
-1,cfn-task-123,2025-10-21T10:30:00Z,agent_spawn,loop3,coder-1-1,1,"{""agent_type"": ""coder"", ""timeout"": 900}",INFO
-```
-
-## AI Agent Consumption
-
-AI agents can query logs to improve workflows:
-
-```bash
-# Example: Analyze agent failures
-ERRORS=$(./query-logs.sh --task-id "cfn-task-123" --level ERROR --format json)
-
-# Parse JSON with jq
-echo "$ERRORS" | jq -r '.[] | "\(.timestamp) [\(.agent_id)] \(.event_type): \(.details)"'
-
-# Example output:
-# 2025-10-21T10:45:00Z [coder-1-1] agent_failure: {"error": "skill_execution_error", "output": "Unknown parameter --invalid-param"}
-```
-
-### Common Queries for AI Analysis
-
-```bash
-# Find agents with highest failure rate
-./query-logs.sh --task-id "$TASK_ID" --event-type agent_failure --format json | \
-  jq -r '.[].agent_id' | sort | uniq -c | sort -nr
-
-# Calculate average agent latency per loop
-./query-logs.sh --task-id "$TASK_ID" --event-type agent_complete --format json | \
-  jq -r '.[] | "\(.loop) \(.details | fromjson | .latency_ms)"' | \
-  awk '{sum[$1]+=$2; count[$1]++} END {for (loop in sum) print loop, sum[loop]/count[loop]}'
-
-# Find parameter errors (for troubleshooting implementations)
-./query-logs.sh --task-id "$TASK_ID" --event-type parameter_error --format json
-
-# Track decision history
-./query-logs.sh --task-id "$TASK_ID" --event-type po_decision --format json | \
-  jq -r '.[] | "\(.iteration): \(.details | fromjson | .decision) - \(.details | fromjson | .reasoning)"'
-```
-
-## Logged Events in Orchestrator
-
-The orchestrator automatically logs:
-
-1. **Line ~643:** Swarm initialization with all configuration
-2. **Line ~811:** Each Loop 3 agent spawn with timeout
-3. **Line ~892:** Each Loop 3 agent completion with confidence and files changed
-4. **Line ~917:** Each Loop 3 agent failure with error details
-5. **Line ~1082:** Gate check failures with consensus scores
-6. **Line ~1115:** Gate check successes
-7. **Line ~1440:** Product Owner decisions with reasoning
-
-All logs include `2>/dev/null || true` to ensure logging failures don't break orchestration.
-
-## Performance
-
-- **Write latency:** ~5-10ms per log entry
-- **Query latency:** ~10-50ms for typical queries (< 1000 events)
-- **Storage:** ~500 bytes per event (compressed SQLite)
-- **Indexes:** Optimized for task_id, event_type, timestamp, level queries
-
-## Debugging
-
-### Check if logging is working
-
-```bash
-# Check database exists
-ls -lh data/cfn-loop.db
-
-# Count total log entries
-sqlite3 data/cfn-loop.db "SELECT COUNT(*) FROM cfn_loop_logs;"
-
-# Get latest 5 events
-./query-logs.sh --task-id "YOUR_TASK_ID" --limit 5 --format table
-```
-
-### Common issues
-
-**Issue:** "Error: Database not found"
-- **Cause:** No logs written yet
-- **Fix:** Run a CFN loop task to generate logs
-
-**Issue:** "Error: --details must be valid JSON"
-- **Cause:** Malformed JSON in details parameter
-- **Fix:** Validate JSON with `echo "$DETAILS" | jq empty`
-
-**Issue:** Logging fails silently
-- **Cause:** `2>/dev/null || true` suppresses errors
-- **Fix:** Remove `2>/dev/null` temporarily to see error messages
-
-## Web Portal Integration
-
-The web portal can query logs for real-time visibility:
-
-```typescript
-// Example: Fetch logs for task
-const logs = await fetch('/api/logs?task_id=cfn-task-123&event_type=agent_spawn');
-const events = await logs.json();
-
-// Display in timeline
-events.forEach(event => {
-  console.log(`${event.timestamp} [${event.loop}] ${event.agent_id}: ${event.event_type}`);
-});
-```
-
-## Retention
-
-- **Default:** Logs persist indefinitely in SQLite
-- **Recommended:** Implement cleanup job to delete logs older than 30 days for non-critical tasks
-- **Critical tasks:** Retain logs for audit trail
-
-```bash
-# Example: Delete logs older than 30 days
-sqlite3 data/cfn-loop.db "DELETE FROM cfn_loop_logs WHERE timestamp < datetime('now', '-30 days');"
-```
-
-## Privacy & Security
-
-- **Sensitive data:** Avoid logging secrets, API keys, or PII in details field
-- **Access control:** Database file permissions (chmod 600) restrict access
-- **Audit trail:** Logs include full decision reasoning for compliance
-
-## Future Enhancements
-
-- [ ] Structured logging levels (DEBUG for verbose agent output)
-- [ ] Log streaming to external systems (Elasticsearch, CloudWatch)
-- [ ] Automatic anomaly detection (high failure rates, long latencies)
-- [ ] Log rotation and archival
-- [ ] Web UI for log browsing and search

package/.claude/skills/cfn-redis-coordination/README.md

@@ -1,65 +0,0 @@
-# Redis Coordination Skill
-
-## Quick Start
-
-### Prerequisites
-- Redis 5.0+
-- bash
-- jq
-- redis-cli
-
-### Installation
-1. Ensure Redis is running
-2. Configure Redis connection in `config.json`
-3. Make scripts executable:
-```bash
-chmod +x invoke-waiting-mode.sh
-```
-
-### Basic Usage
-
-#### Consensus Collection (Updated)
-```bash
-# Agent reports results
-./invoke-waiting-mode.sh report \
-  --task-id "my-task" \
-  --agent-id "agent-1" \
-  --confidence 0.95
-
-# Collect and evaluate consensus
-./invoke-waiting-mode.sh collect \
-  --task-id "my-task" \
-  --agent-ids "agent-1,agent-2,agent-3"
-```
-
-## Important Changes in P7 (Redis Script Cleanup)
-
-### Deprecation Notices
-- 🚨 `enter` and `wake` subcommands are NO LONGER SUPPORTED
-- Agents should exit cleanly without waiting mode
-- Coordinator spawns agents directly
-- Fork-ID references have been removed
-
-### Migration Guide
-- Update agent scripts to exit cleanly after task
-- Remove manual waiting mode calls
-- Use direct agent spawning in orchestrator
-
-## Script Categories
-- **Production Scripts**:
-  - `invoke-waiting-mode.sh`: Redis coordination wrapper
-  - `orchestrate-cfn-loop.sh`: CFN Loop orchestration
-- **Demos and Tests**: Located in `./demos/`
-
-## Performance
-- Zero-token waiting
-- Sub-100ms wake-up latency
-- Supports 10+ concurrent agents
-- Configurable consensus thresholds
-
-## Configuration Options
-See `SKILL.md` for detailed configuration and usage instructions.
-
-## Troubleshooting
-- If you encounter issues with old scripts, refer to migration guide
-- Test scripts are available in `./demos/` directory

package/.claude/skills/cfn-redis-coordination/SECURITY_REVIEW.md

@@ -1,25 +0,0 @@
-# Security Review: Metrics Export System (Phase 7)
-
-## Overview
-Security review conducted by security-specialist-3 for Redis Coordination metrics export functionality.
-
-## Confidence Score: 0.92 (High)
-
-### Key Findings
-- ✅ Robust input validation
-- ✅ Secure file handling
-- ✅ Minimal data exposure
-- ⚠️ Recommended ACL improvements
-
-### Recommendations
-1. Implement optional PII sanitization
-2. Enhance Redis key access controls
-3. Create metrics export audit logging
-4. Add optional export encryption
-
-### Compliance
-- NIST SP 800-53 Alignment: Moderate Impact
-- SOC 2 Type II Ready
-- GDPR Data Minimization Compliant
-
-Full detailed report available in source code comments.