claude-flow-novice 2.9.1 → 2.10.0

package/.claude/skills/cfn-redis-coordination/SKILL.md

@@ -0,0 +1,720 @@
+# Redis Coordination Skill
+
+**Version:** 3.0.0 (Pure Coordination Primitives)
+**Last Updated:** 2025-10-23
+
+**⚠️ DEPRECATION NOTICE:**
+The monolithic `orchestrate-cfn-loop.sh` (70KB) has been replaced by the modular CFN v3 orchestrator at `.claude/skills/cfn-loop-orchestration/orchestrate.sh` (26KB). See CFN v3 documentation for details.
+
+---
+
+## Overview
+
+The Redis Coordination Skill provides low-level coordination primitives for distributed agent orchestration. This skill is framework-agnostic and supports any coordination pattern (swarm, mesh, hierarchical).
+
+**Key Capabilities:**
+- Generic JSON context storage and retrieval
+- Pub/sub signaling (LPUSH/BLPOP)
+- Agent result collection and aggregation
+- Zero-token waiting mechanisms
+- Consensus calculation
+
+**Design Philosophy:**
+- Pure primitives, no workflow-specific logic
+- Minimal assumptions about coordination patterns
+- Clean, composable interfaces
+- Framework-agnostic implementations
+
+---
+
+## Core Primitives
+
+### 1. Context Storage (`store-context.sh`)
+
+Store arbitrary JSON context in Redis with automatic TTL management.
+
+**Usage:**
+```bash
+./store-context.sh \
+  --task-id "unique-task-123" \
+  --key "epic-context" \
+  --value '{"goal": "Build feature X", "scope": ["A", "B"]}' \
+  --ttl 86400
+```
+
+**Parameters:**
+- `--task-id` (required): Unique task identifier
+- `--key` (required): Context key name
+- `--value` (required): Valid JSON string
+- `--ttl` (optional): Time-to-live in seconds (default: 86400 = 24h)
+- `--namespace` (optional): Redis key namespace (default: "context")
+
+**Returns:**
+- Redis key on success: `context:unique-task-123:epic-context`
+- Exit code 0 on success, 1 on error
+
+**Redis Key Format:**
+```
+{namespace}:{task_id}:{key}
+  └─ Hash fields:
+     - value: JSON payload
+     - metadata: {stored_at, ttl_seconds}
+```
+
+**Example:**
+```bash
+# Store epic context
+REDIS_KEY=$(./store-context.sh \
+  --task-id "task-001" \
+  --key "epic" \
+  --value '{"goal": "Authentication system"}')
+
+echo "Stored at: $REDIS_KEY"
+# Output: Stored at: context:task-001:epic
+```
+
+---
+
+### 2. Context Retrieval (`retrieve-context.sh`)
+
+Retrieve JSON context from Redis with optional metadata.
+
+**Usage:**
+```bash
+./retrieve-context.sh \
+  --task-id "unique-task-123" \
+  --key "epic-context"
+```
+
+**Parameters:**
+- `--task-id` (required): Unique task identifier
+- `--key` (required): Context key name
+- `--namespace` (optional): Redis key namespace (default: "context")
+- `--with-metadata` (optional): Include storage metadata in output
+
+**Returns:**
+- JSON value on success
+- Exit code 0 on success, 1 if key not found or invalid
+
+**Example:**
+```bash
+# Retrieve context
+CONTEXT=$(./retrieve-context.sh \
+  --task-id "task-001" \
+  --key "epic")
+
+echo "$CONTEXT" | jq .
+# Output: {"goal": "Authentication system"}
+
+# With metadata
+./retrieve-context.sh \
+  --task-id "task-001" \
+  --key "epic" \
+  --with-metadata | jq .
+# Output:
+# {
+#   "value": {"goal": "Authentication system"},
+#   "metadata": {"stored_at": "2025-10-23T10:30:00Z", "ttl_seconds": 86400},
+#   "ttl_remaining": 85000
+# }
+```
+
+---
+
+### 3. Signaling (`signal.sh`)
+
+Generic pub/sub signaling for agent coordination using LPUSH/BLPOP.
+
+**Commands:**
+- `send`: Send signal to a queue (LPUSH)
+- `wait`: Wait for signal with timeout (BLPOP)
+- `broadcast`: Send signal to multiple agents
+
+**Usage - Send:**
+```bash
+./signal.sh send \
+  --task-id "task-001" \
+  --signal "gate-passed" \
+  --payload '{"threshold": 0.75, "passed": true}'
+```
+
+**Usage - Wait:**
+```bash
+# Wait indefinitely (timeout=0)
+PAYLOAD=$(./signal.sh wait \
+  --task-id "task-001" \
+  --signal "gate-passed")
+
+# Wait with timeout (120 seconds)
+PAYLOAD=$(./signal.sh wait \
+  --task-id "task-001" \
+  --signal "gate-passed" \
+  --timeout 120)
+```
+
+**Usage - Broadcast:**
+```bash
+./signal.sh broadcast \
+  --task-id "task-001" \
+  --signal "iteration-start" \
+  --agents "agent-1,agent-2,agent-3" \
+  --payload '{"iteration": 2, "reason": "improve_quality"}'
+```
+
+**Parameters:**
+- `--task-id` (required): Unique task identifier
+- `--signal` (required): Signal name
+- `--payload` (optional): JSON payload (default: `{}`)
+- `--timeout` (optional): Wait timeout in seconds (default: 0 = infinite)
+- `--agents` (required for broadcast): Comma-separated agent IDs
+- `--namespace` (optional): Redis key namespace (default: "signal")
+
+**Redis Key Formats:**
+```
+# Send/Wait:
+signal:{task_id}:{signal_name}
+
+# Broadcast:
+signal:{task_id}:{agent_id}:{signal_name}
+```
+
+**Example - Simple Coordination:**
+```bash
+# Agent 1: Wait for signal
+(
+  ./signal.sh wait --task-id "task-001" --signal "start-work"
+  echo "Received start signal, beginning work..."
+) &
+
+# Agent 2: Send signal after preparation
+sleep 2
+./signal.sh send --task-id "task-001" --signal "start-work" --payload '{}'
+```
+
+---
+
+### 4. Result Collection (`collect-results.sh`)
+
+Collect and aggregate agent results with confidence scoring and consensus calculation.
+
+**Usage:**
+```bash
+./collect-results.sh \
+  --task-id "task-001" \
+  --agent-ids "coder-1,reviewer-1,tester-1"
+```
+
+**Parameters:**
+- `--task-id` (required): Unique task identifier
+- `--agent-ids` (required): Comma-separated agent IDs
+- `--namespace` (optional): Redis key namespace (default: "result")
+- `--calculate-consensus` (optional): Calculate average confidence score
+- `--min-confidence` (optional): Minimum confidence threshold
+- `--include-metadata` (optional): Include agent metadata in output
+- `--timeout` (optional): Wait timeout per agent in seconds (default: 0)
+
+**Expected Redis Structure (Per Agent):**
+```
+result:{task_id}:{agent_id}
+  └─ Hash fields:
+     - confidence: 0.85
+     - iteration: 1
+     - timestamp: 2025-10-23T10:30:00Z
+     - output: "Implementation complete"
+```
+
+**Returns JSON:**
+```json
+{
+  "task_id": "task-001",
+  "total_agents": 3,
+  "successful_agents": 3,
+  "failed_agents": [],
+  "results": [
+    {
+      "agent_id": "coder-1",
+      "confidence": 0.92,
+      "iteration": 1,
+      "timestamp": "2025-10-23T10:30:00Z",
+      "output": "Implementation complete"
+    },
+    {
+      "agent_id": "reviewer-1",
+      "confidence": 0.88,
+      "iteration": 1,
+      "timestamp": "2025-10-23T10:31:00Z",
+      "output": "Code review passed"
+    }
+  ],
+  "consensus": 0.90
+}
+```
+
+**Example - Calculate Consensus:**
+```bash
+# Collect results with consensus calculation
+RESULTS=$(./collect-results.sh \
+  --task-id "task-001" \
+  --agent-ids "coder-1,reviewer-1,tester-1" \
+  --calculate-consensus \
+  --min-confidence 0.75)
+
+CONSENSUS=$(echo "$RESULTS" | jq -r '.consensus')
+echo "Team consensus: $CONSENSUS"
+
+# Check if consensus meets threshold
+if (( $(echo "$CONSENSUS >= 0.90" | bc -l) )); then
+  echo "✅ Consensus threshold met"
+else
+  echo "❌ Needs iteration"
+fi
+```
+
+---
+
+## Coordination Patterns
+
+### Pattern 1: Simple Chain (Sequential)
+
+Agents execute sequentially, each waiting for the previous to complete.
+
+```bash
+TASK_ID="task-001"
+
+# Agent 1: Execute and signal completion
+(
+  echo "Agent 1: Working..."
+  sleep 2
+  ./signal.sh send --task-id "$TASK_ID" --signal "agent-1-done" --payload '{}'
+) &
+
+# Agent 2: Wait for Agent 1, then execute
+(
+  ./signal.sh wait --task-id "$TASK_ID" --signal "agent-1-done"
+  echo "Agent 2: Working..."
+  sleep 2
+  ./signal.sh send --task-id "$TASK_ID" --signal "agent-2-done" --payload '{}'
+) &
+
+# Wait for all agents
+wait
+echo "Chain complete"
+```
+
+### Pattern 2: Broadcast with Consensus
+
+Coordinator broadcasts work to multiple agents, then collects consensus.
+
+```bash
+TASK_ID="task-002"
+AGENTS="agent-1,agent-2,agent-3"
+
+# Store context for agents
+./store-context.sh \
+  --task-id "$TASK_ID" \
+  --key "work-context" \
+  --value '{"task": "Review code", "file": "main.py"}'
+
+# Broadcast start signal to all agents
+./signal.sh broadcast \
+  --task-id "$TASK_ID" \
+  --signal "start-work" \
+  --agents "$AGENTS" \
+  --payload '{"iteration": 1}'
+
+# Wait for agents to complete (agents must report results to Redis)
+sleep 10
+
+# Collect results and calculate consensus
+RESULTS=$(./collect-results.sh \
+  --task-id "$TASK_ID" \
+  --agent-ids "$AGENTS" \
+  --calculate-consensus \
+  --min-confidence 0.75)
+
+echo "$RESULTS" | jq .
+```
+
+### Pattern 3: Hierarchical (Loop Dependencies)
+
+Implement multi-stage workflows with gate checks between stages.
+
+```bash
+TASK_ID="task-003"
+
+# Stage 1: Implementation agents
+LOOP3_AGENTS="coder-1,researcher-1"
+./signal.sh broadcast \
+  --task-id "$TASK_ID" \
+  --signal "stage-1-start" \
+  --agents "$LOOP3_AGENTS" \
+  --payload '{}'
+
+# Wait for implementation results
+sleep 10
+LOOP3_RESULTS=$(./collect-results.sh \
+  --task-id "$TASK_ID" \
+  --agent-ids "$LOOP3_AGENTS" \
+  --calculate-consensus)
+
+LOOP3_CONSENSUS=$(echo "$LOOP3_RESULTS" | jq -r '.consensus')
+
+# Gate check
+GATE_THRESHOLD=0.75
+if (( $(echo "$LOOP3_CONSENSUS >= $GATE_THRESHOLD" | bc -l) )); then
+  echo "✅ Gate passed: $LOOP3_CONSENSUS"
+
+  # Stage 2: Validation agents (only if gate passed)
+  LOOP2_AGENTS="reviewer-1,tester-1"
+  ./signal.sh send \
+    --task-id "$TASK_ID" \
+    --signal "gate-passed" \
+    --payload "{\"loop3_consensus\": $LOOP3_CONSENSUS}"
+
+  ./signal.sh broadcast \
+    --task-id "$TASK_ID" \
+    --signal "stage-2-start" \
+    --agents "$LOOP2_AGENTS" \
+    --payload '{}'
+else
+  echo "❌ Gate failed: $LOOP3_CONSENSUS - relaunch Stage 1"
+fi
+```
+
+---
+
+## Agent Integration Examples
+
+### Example 1: Agent Storing Context
+
+```bash
+#!/bin/bash
+# Agent: backend-dev
+
+TASK_ID="$1"
+AGENT_ID="backend-dev-1"
+
+# Retrieve work context
+CONTEXT=$(./retrieve-context.sh --task-id "$TASK_ID" --key "work-context")
+GOAL=$(echo "$CONTEXT" | jq -r '.goal')
+
+echo "Working on: $GOAL"
+
+# Do work...
+sleep 5
+
+# Report confidence to Redis
+redis-cli HSET "result:${TASK_ID}:${AGENT_ID}" \
+  confidence 0.92 \
+  iteration 1 \
+  timestamp "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
+  output "Implementation complete"
+
+echo "Reported confidence: 0.92"
+```
+
+### Example 2: Agent Waiting for Signal
+
+```bash
+#!/bin/bash
+# Agent: validator
+
+TASK_ID="$1"
+AGENT_ID="validator-1"
+
+echo "Waiting for gate to pass..."
+
+# Block until signal received
+SIGNAL_PAYLOAD=$(./signal.sh wait \
+  --task-id "$TASK_ID" \
+  --signal "gate-passed" \
+  --timeout 300)
+
+if [ $? -eq 0 ]; then
+  LOOP3_CONSENSUS=$(echo "$SIGNAL_PAYLOAD" | jq -r '.loop3_consensus')
+  echo "Gate passed with consensus: $LOOP3_CONSENSUS"
+
+  # Retrieve implementation results
+  CONTEXT=$(./retrieve-context.sh --task-id "$TASK_ID" --key "implementation")
+
+  # Validate...
+  echo "Validating implementation..."
+  sleep 3
+
+  # Report result
+  redis-cli HSET "result:${TASK_ID}:${AGENT_ID}" \
+    confidence 0.88 \
+    iteration 1 \
+    timestamp "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
+    output "Validation passed"
+else
+  echo "Timeout waiting for gate signal"
+  exit 1
+fi
+```
+
+---
+
+## Deprecated Features (v3.0.0)
+
+### Waiting Mode (invoke-waiting-mode.sh)
+
+**Status:** Partially deprecated (report/collect still active)
+
+The `enter` and `wake` subcommands are deprecated as they caused agent lifecycle issues (indefinite blocking). Agents should now exit cleanly after reporting results.
+
+**Still Supported:**
+- `report`: Report confidence scores
+- `collect`: Collect consensus
+- `shutdown`: Clean shutdown
+
+**No Longer Supported:**
+- `enter`: Agents should exit instead
+- `wake`: Use broadcast signaling instead
+
+**Migration Path:**
+```bash
+# OLD (deprecated):
+./invoke-waiting-mode.sh enter --task-id "$TASK_ID" --agent-id "$AGENT_ID"
+
+# NEW (use signal.sh):
+./signal.sh wait --task-id "$TASK_ID" --signal "wake-signal"
+```
+
+### CFN Loop Orchestrator (orchestrate-cfn-loop.sh)
+
+**Status:** Deprecated (kept for backward compatibility)
+
+The orchestrator script contains CFN-specific workflow logic and will be moved to the `cfn-loop-validation` skill in a future release.
+
+**Recommendation:** Build custom orchestrators using the core primitives instead of relying on this monolithic script.
+
+---
+
+## Best Practices
+
+### 1. Always Set TTL
+
+Context and signals should have reasonable TTLs to prevent Redis memory leaks:
+
+```bash
+# Good: 24-hour TTL
+./store-context.sh --task-id "$TASK_ID" --key "context" --value "$JSON" --ttl 86400
+
+# Signals auto-expire after 1 hour
+./signal.sh send --task-id "$TASK_ID" --signal "start" --payload '{}'
+```
+
+### 2. Validate JSON Before Storage
+
+```bash
+# Validate JSON structure
+if echo "$CONTEXT_JSON" | jq empty 2>/dev/null; then
+  ./store-context.sh --task-id "$TASK_ID" --key "context" --value "$CONTEXT_JSON"
+else
+  echo "Error: Invalid JSON"
+  exit 1
+fi
+```
+
+### 3. Use Namespaces for Isolation
+
+```bash
+# Separate production and test environments
+./store-context.sh \
+  --task-id "$TASK_ID" \
+  --key "config" \
+  --value "$JSON" \
+  --namespace "prod:context"
+
+./store-context.sh \
+  --task-id "$TASK_ID" \
+  --key "config" \
+  --value "$JSON" \
+  --namespace "test:context"
+```
+
+### 4. Handle Timeouts Gracefully
+
+```bash
+# Always check signal.sh exit code
+if SIGNAL=$(./signal.sh wait --task-id "$TASK_ID" --signal "ready" --timeout 60); then
+  echo "Signal received: $SIGNAL"
+else
+  echo "Timeout - proceeding with default behavior"
+fi
+```
+
+### 5. Clean Up After Task Completion
+
+```bash
+# Delete task-specific keys after completion
+redis-cli DEL "context:${TASK_ID}:*"
+redis-cli DEL "signal:${TASK_ID}:*"
+redis-cli DEL "result:${TASK_ID}:*"
+```
+
+---
+
+## Testing
+
+### Unit Tests
+
+Each primitive has corresponding tests in `tests/primitives/`:
+
+```bash
+# Test context storage
+./tests/primitives/test-store-context.sh
+
+# Test context retrieval
+./tests/primitives/test-retrieve-context.sh
+
+# Test signaling
+./tests/primitives/test-signal.sh
+
+# Test result collection
+./tests/primitives/test-collect-results.sh
+```
+
+### Integration Tests
+
+Test coordination patterns end-to-end:
+
+```bash
+# Test chain pattern
+./tests/integration/test-chain-pattern.sh
+
+# Test broadcast pattern
+./tests/integration/test-broadcast-pattern.sh
+
+# Test hierarchical pattern
+./tests/integration/test-hierarchical-pattern.sh
+```
+
+---
+
+## Performance
+
+### Benchmarks (100 operations)
+
+| Operation | Avg Latency | Throughput |
+|-----------|-------------|------------|
+| store-context.sh | 3ms | 333 ops/s |
+| retrieve-context.sh | 2ms | 500 ops/s |
+| signal.sh send | 2ms | 500 ops/s |
+| signal.sh wait (immediate) | 3ms | 333 ops/s |
+| collect-results.sh (3 agents) | 8ms | 125 ops/s |
+
+**Zero-Token Waiting:**
+- BLPOP blocks without API calls (0 token cost)
+- Immediate wake-up (<100ms latency)
+- Scales to 100+ concurrent agents
+
+---
+
+## Security Considerations
+
+### 1. No Sensitive Data in Payloads
+
+Never store API keys, credentials, or secrets in Redis context:
+
+```bash
+# BAD:
+./store-context.sh --key "config" --value '{"api_key": "sk-xxx"}'
+
+# GOOD:
+./store-context.sh --key "config" --value '{"api_key_ref": "env:API_KEY"}'
+```
+
+### 2. Namespace Isolation
+
+Use namespaces to prevent cross-task contamination:
+
+```bash
+--namespace "user-${USER_ID}:context"
+```
+
+### 3. TTL Enforcement
+
+Always set TTLs to prevent indefinite data retention:
+
+```bash
+--ttl 3600  # 1 hour maximum
+```
+
+---
+
+## Error Handling
+
+### Common Errors
+
+**Error:** `Context key does not exist`
+```bash
+# Check if key exists before retrieval
+if redis-cli EXISTS "context:${TASK_ID}:${KEY}" | grep -q "1"; then
+  ./retrieve-context.sh --task-id "$TASK_ID" --key "$KEY"
+else
+  echo "Key not found, using default"
+fi
+```
+
+**Error:** `Timeout waiting for signal`
+```bash
+# Always provide timeout for production
+./signal.sh wait --task-id "$TASK_ID" --signal "start" --timeout 300 || {
+  echo "Timeout - sending fallback signal"
+  ./signal.sh send --task-id "$TASK_ID" --signal "timeout-fallback" --payload '{}'
+}
+```
+
+**Error:** `Failed to retrieve context value`
+```bash
+# Validate Redis connection
+if ! redis-cli PING | grep -q "PONG"; then
+  echo "Redis unavailable"
+  exit 1
+fi
+```
+
+---
+
+## Version History
+
+### v3.0.0 (2025-10-23) - Pure Coordination Primitives
+- **Breaking:** Removed CFN-specific logic from primitives
+- **Added:** `store-context.sh` - Generic JSON storage
+- **Added:** `retrieve-context.sh` - Generic JSON retrieval
+- **Added:** `signal.sh` - Pub/sub signaling
+- **Added:** `collect-results.sh` - Result aggregation
+- **Deprecated:** `invoke-waiting-mode.sh` enter/wake commands
+- **Deprecated:** `orchestrate-cfn-loop.sh` (moved to cfn-loop-validation skill)
+- **Migration:** CFN-specific logic moved to separate orchestration layer
+
+### v2.1.0 (2025-10-20) - Agent Completion Protocol
+- Three-layer timeout protection
+- Process-based completion detection
+- Heartbeat monitoring
+
+### v2.0.0 (2025-10-18) - Metrics & Observability
+- Comprehensive metrics collection
+- Prometheus export support
+- Deliverable verification (BUG #11 fix)
+
+---
+
+## Support
+
+**Skill Owner:** Redis Coordination Team
+**Documentation:** `.claude/skills/redis-coordination/SKILL.md`
+**Issues:** Tag with `skill:redis-coordination`
+
+**Related Skills:**
+- `cfn-loop-validation` - CFN Loop orchestration (uses these primitives)
+- `agent-spawning` - Agent lifecycle management
+- `hook-pipeline` - Post-edit validation
+
+---
+
+## License
+
+Part of Claude Flow Novice - AI Agent Orchestration Framework