claude-flow-novice 2.5.2 → 2.8.1

package/.claude/skills/redis-coordination/AGENT_LOGGING.md

@@ -0,0 +1,280 @@
+# Agent Logging System
+
+Dual-output logging for CLI-spawned agents: terminal + Redis + portal visibility.
+
+## Features
+
+✅ **Terminal output** - Colored logs for debugging in tmux/terminal
+✅ **File logging** - Persisted to `/tmp/agent-*.log`
+✅ **Redis pub/sub** - Real-time streaming to web portal
+✅ **Redis history** - Last 7 days stored in sorted sets
+✅ **Portal filters** - Filter by repository, agent, or log level
+
+## Quick Start
+
+### 1. Source in Agent Scripts
+
+```bash
+#!/usr/bin/env bash
+source .claude/skills/redis-coordination/agent-log.sh
+
+# Set context
+export AGENT_ID="backend-dev-1"
+export TASK_ID="$1"  # Pass from command line
+
+# Log throughout execution
+agent_log "info" "Starting task execution"
+agent_log "debug" "Loading configuration from $CONFIG_FILE"
+agent_log "warn" "High memory usage detected: ${MEMORY}MB"
+agent_log "error" "Connection failed: $ERROR_MSG"
+```
+
+### 2. Standalone Usage
+
+```bash
+# Direct command-line usage
+./.claude/skills/redis-coordination/agent-log.sh info "Agent initialized" \
+  --agent-id researcher-1 \
+  --task-id task-auth-123
+
+# From any script
+./.claude/skills/redis-coordination/agent-log.sh error "Validation failed" \
+  --agent-id reviewer-2 \
+  --task-id task-auth-123
+```
+
+### 3. Log Levels
+
+- **debug** - Detailed debugging information (cyan)
+- **info** - General informational messages (green)
+- **warn** - Warning messages (yellow)
+- **error** - Error conditions (red)
+
+## Output Destinations
+
+**1. Terminal (with colors)**
+```
+[INFO] [backend-dev-1] Starting authentication implementation
+[WARN] [backend-dev-1] Database connection slow
+[ERROR] [reviewer-1] JWT validation failed
+```
+
+**2. File (`/tmp/agent-<id>.log`)**
+```
+[2025-10-20T07:02:48Z] [INFO] [backend-dev-1] Starting authentication implementation
+[2025-10-20T07:02:48Z] [WARN] [backend-dev-1] Database connection slow
+[2025-10-20T07:02:48Z] [ERROR] [reviewer-1] JWT validation failed
+```
+
+**3. Redis (`swarm:<task-id>:logs`)**
+```json
+{
+  "level": "info",
+  "message": "Starting authentication implementation",
+  "agentId": "backend-dev-1",
+  "taskId": "task-auth-123",
+  "repository": "claude-flow-novice",
+  "timestamp": "2025-10-20T07:02:48Z"
+}
+```
+
+**4. Web Portal (http://localhost:3456 → Logs tab)**
+- Real-time streaming via WebSocket
+- Filter by repository, agent, or level
+- Shows last 1000 logs (100 displayed)
+- Auto-refreshes on new logs
+
+## Integration Points
+
+### Option 1: Source in Main Agent Script
+
+```bash
+# In your main agent script
+source .claude/skills/redis-coordination/agent-log.sh
+
+# Set global context once
+export AGENT_ID="${AGENT_TYPE}-${AGENT_NUM}"
+export TASK_ID="$TASK_ID"
+
+# Then log freely
+agent_log "info" "Agent spawned successfully"
+# ... agent work ...
+agent_log "info" "Task completed with confidence 0.95"
+```
+
+### Option 2: Wrap Agent Spawning
+
+```bash
+# In orchestrate-cfn-loop.sh or spawn-agent.sh
+./.claude/skills/redis-coordination/agent-log.sh info \
+  "Spawning agent: $AGENT_ID for task: $TASK_ID" \
+  --agent-id "$AGENT_ID" \
+  --task-id "$TASK_ID"
+
+# Spawn agent
+npx claude-flow-novice agent "$AGENT_TYPE" --task-id "$TASK_ID"
+
+# Log completion
+./.claude/skills/redis-coordination/agent-log.sh info \
+  "Agent $AGENT_ID completed" \
+  --agent-id "$AGENT_ID" \
+  --task-id "$TASK_ID"
+```
+
+### Option 3: Add to Existing Agent Logic
+
+If you have existing CLI agent code, wrap key operations:
+
+```bash
+# Before long operation
+agent_log "info" "Starting database migration"
+
+# During operation
+if [ "$ROWS_UPDATED" -gt 10000 ]; then
+  agent_log "warn" "Large migration in progress: $ROWS_UPDATED rows"
+fi
+
+# On error
+if ! run_migration; then
+  agent_log "error" "Migration failed: $ERROR"
+  exit 1
+fi
+
+# On success
+agent_log "info" "Migration completed: $ROWS_UPDATED rows updated"
+```
+
+## Advanced Options
+
+### Skip Terminal Output (Quiet Mode)
+
+```bash
+agent_log "info" "Background task started" --no-terminal
+```
+
+### Skip Redis Output (Local Only)
+
+```bash
+agent_log "debug" "Local debug info" --no-redis
+```
+
+### Manual Context Override
+
+```bash
+agent_log "error" "Connection timeout" \
+  --agent-id "custom-agent" \
+  --task-id "override-task-id"
+```
+
+## Portal Usage
+
+1. **Open portal:** http://localhost:3456
+2. **Go to Logs tab** (📝 Logs)
+3. **Filter logs:**
+   - Repository: Select specific repo or "All Repositories"
+   - Agent: Select specific agent or "All Agents"
+   - Level: debug/info/warn/error or "All Levels"
+4. **Clear logs:** Click "Clear Logs" button
+
+**Stats shown:**
+- Total logs in memory
+- Number of repositories logging
+- Number of active agents
+
+## Redis Keys
+
+**Pub/Sub Channel:**
+```
+swarm:<task-id>:logs
+```
+
+**Sorted Set (History):**
+```
+swarm:<task-id>:logs:history
+```
+- Score: Unix timestamp
+- Value: JSON log entry
+- TTL: 7 days (604800 seconds)
+
+## Examples
+
+### Example 1: CFN Loop Agent
+
+```bash
+#!/usr/bin/env bash
+source .claude/skills/redis-coordination/agent-log.sh
+
+AGENT_ID="coder-3"
+TASK_ID="$1"
+
+agent_log "info" "Starting Loop 3 implementation phase"
+
+# Implement feature
+agent_log "debug" "Reading specification from spec.md"
+agent_log "info" "Implementing authentication service"
+
+if ! run_tests; then
+  agent_log "error" "Tests failed - iteration needed"
+  exit 1
+fi
+
+agent_log "info" "Implementation complete - confidence: 0.88"
+```
+
+### Example 2: Orchestrator Logging
+
+```bash
+# In orchestrate-cfn-loop.sh
+agent_log "info" "CFN Loop started - mode: $MODE" \
+  --agent-id "orchestrator" \
+  --task-id "$TASK_ID"
+
+agent_log "info" "Spawning Loop 3 agents: $LOOP3_AGENTS" \
+  --agent-id "orchestrator" \
+  --task-id "$TASK_ID"
+
+# ... spawn agents ...
+
+agent_log "info" "Loop 3 consensus: $CONSENSUS (gate: $GATE_THRESHOLD)" \
+  --agent-id "orchestrator" \
+  --task-id "$TASK_ID"
+```
+
+## Troubleshooting
+
+**Logs not appearing in portal:**
+1. Check Redis connection: `redis-cli ping`
+2. Check portal subscription: Look for "✅ Subscribed to swarm:*:logs" in portal startup
+3. Verify task ID matches swarm task ID
+4. Check Redis pub/sub: `redis-cli psubscribe "swarm:*:logs"`
+
+**Colors not showing in terminal:**
+- WSL/Linux should show colors automatically
+- tmux: Colors should work by default
+- If colors missing, terminal may not support ANSI codes
+
+**Log file not created:**
+- Check permissions on `/tmp/`
+- Verify `AGENT_ID` is set (file named `/tmp/agent-${AGENT_ID}.log`)
+
+## Performance
+
+- **Minimal overhead:** ~5ms per log entry
+- **Non-blocking:** Redis operations use `> /dev/null 2>&1 || true`
+- **Memory efficient:** Portal keeps max 1000 logs, shows 100
+- **Auto-cleanup:** Redis history expires after 7 days
+
+## Migration from Old Logging
+
+**Before:**
+```bash
+echo "[INFO] Agent started"
+```
+
+**After:**
+```bash
+source .claude/skills/redis-coordination/agent-log.sh
+agent_log "info" "Agent started"
+```
+
+Benefits: Same terminal output + Redis pub/sub + portal visibility + file logging

package/.claude/skills/redis-coordination/LOGGING.md

@@ -0,0 +1,260 @@
+# CFN Loop Logging System
+
+## Overview
+
+The CFN Loop logging system provides comprehensive visibility into agent execution, decisions, and errors. All logs are stored in SQLite for efficient querying and analysis by AI agents.
+
+## Database Location
+
+```bash
+Default: claude-flow-novice/data/cfn-loop.db
+Custom:  Set DB_PATH environment variable
+```
+
+## Schema
+
+```sql
+CREATE TABLE cfn_loop_logs (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  task_id TEXT NOT NULL,              -- Task/swarm identifier
+  timestamp TEXT DEFAULT (datetime('now')),  -- ISO 8601 timestamp
+  event_type TEXT NOT NULL,           -- Event category
+  loop TEXT,                          -- loop3, loop2, product_owner, coordinator
+  agent_id TEXT,                      -- Agent identifier (e.g., coder-1-1)
+  iteration INTEGER,                  -- CFN loop iteration number
+  details TEXT,                       -- JSON payload with event-specific data
+  level TEXT DEFAULT 'INFO'           -- DEBUG, INFO, WARN, ERROR
+);
+
+-- Indexes for fast queries
+CREATE INDEX idx_task_id ON cfn_loop_logs(task_id);
+CREATE INDEX idx_event_type ON cfn_loop_logs(event_type);
+CREATE INDEX idx_timestamp ON cfn_loop_logs(timestamp);
+CREATE INDEX idx_level ON cfn_loop_logs(level);
+```
+
+## Event Types
+
+| Event Type | Description | Level | Details Payload |
+|-----------|-------------|-------|-----------------|
+| `swarm_init` | CFN loop initialization | INFO | `{mode, loop3_agents, loop2_agents, product_owner, max_iterations, gate_threshold, consensus_threshold}` |
+| `agent_spawn` | Agent process started | INFO | `{agent_type, timeout}` |
+| `agent_complete` | Agent successfully completed | INFO | `{confidence, confidence_source, files_changed, latency_ms}` |
+| `agent_failure` | Agent execution failed | ERROR | `{error, output}` |
+| `gate_check` | Loop 3 gate validation | INFO/WARN | `{consensus, threshold, result: PASS\|FAIL, decision?}` |
+| `po_decision` | Product Owner strategic decision | INFO | `{decision: PROCEED\|ITERATE\|ABORT, reasoning, confidence}` |
+| `parameter_error` | Invalid parameters passed to tool | ERROR | `{error, command}` |
+
+## Usage
+
+### Logging Events (Orchestrator/Scripts)
+
+```bash
+# Log swarm initialization
+./.claude/skills/redis-coordination/log-event.sh \
+  --task-id "cfn-task-123" \
+  --event-type "swarm_init" \
+  --details '{"mode": "standard", "loop3_agents": "coder", "loop2_agents": "reviewer"}' \
+  --level "INFO"
+
+# Log agent spawn
+./.claude/skills/redis-coordination/log-event.sh \
+  --task-id "cfn-task-123" \
+  --event-type "agent_spawn" \
+  --loop "loop3" \
+  --agent-id "coder-1-1" \
+  --iteration 1 \
+  --details '{"agent_type": "coder", "timeout": 900}' \
+  --level "INFO"
+
+# Log error
+./.claude/skills/redis-coordination/log-event.sh \
+  --task-id "cfn-task-123" \
+  --event-type "agent_failure" \
+  --loop "loop3" \
+  --agent-id "coder-1-1" \
+  --iteration 1 \
+  --details '{"error": "timeout", "output": "Agent exceeded 900s timeout"}' \
+  --level "ERROR"
+```
+
+### Querying Logs (AI Agents/Debugging)
+
+```bash
+# Get all logs for a task
+./query-logs.sh --task-id "cfn-task-123"
+
+# Get only errors
+./query-logs.sh --task-id "cfn-task-123" --level ERROR
+
+# Get Loop 3 agent spawns
+./query-logs.sh --task-id "cfn-task-123" --event-type agent_spawn --loop loop3
+
+# Get Product Owner decisions
+./query-logs.sh --task-id "cfn-task-123" --event-type po_decision
+
+# Get latest 10 events in table format
+./query-logs.sh --task-id "cfn-task-123" --limit 10 --format table
+
+# Get events for specific iteration
+./query-logs.sh --task-id "cfn-task-123" --iteration 2
+
+# Get events for specific agent
+./query-logs.sh --task-id "cfn-task-123" --agent-id "coder-1-1"
+```
+
+### Output Formats
+
+**JSON (default):**
+```json
+[
+  {
+    "id": 1,
+    "task_id": "cfn-task-123",
+    "timestamp": "2025-10-21T10:30:00Z",
+    "event_type": "agent_spawn",
+    "loop": "loop3",
+    "agent_id": "coder-1-1",
+    "iteration": 1,
+    "details": "{\"agent_type\": \"coder\", \"timeout\": 900}",
+    "level": "INFO"
+  }
+]
+```
+
+**Table:**
+```
+id  task_id        timestamp             event_type     loop    agent_id     iteration  level
+1   cfn-task-123   2025-10-21T10:30:00Z  agent_spawn   loop3   coder-1-1    1          INFO
+2   cfn-task-123   2025-10-21T10:45:00Z  agent_complete loop3   coder-1-1    1          INFO
+```
+
+**CSV:**
+```csv
+1,cfn-task-123,2025-10-21T10:30:00Z,agent_spawn,loop3,coder-1-1,1,"{""agent_type"": ""coder"", ""timeout"": 900}",INFO
+```
+
+## AI Agent Consumption
+
+AI agents can query logs to improve workflows:
+
+```bash
+# Example: Analyze agent failures
+ERRORS=$(./query-logs.sh --task-id "cfn-task-123" --level ERROR --format json)
+
+# Parse JSON with jq
+echo "$ERRORS" | jq -r '.[] | "\(.timestamp) [\(.agent_id)] \(.event_type): \(.details)"'
+
+# Example output:
+# 2025-10-21T10:45:00Z [coder-1-1] agent_failure: {"error": "skill_execution_error", "output": "Unknown parameter --invalid-param"}
+```
+
+### Common Queries for AI Analysis
+
+```bash
+# Find agents with highest failure rate
+./query-logs.sh --task-id "$TASK_ID" --event-type agent_failure --format json | \
+  jq -r '.[].agent_id' | sort | uniq -c | sort -nr
+
+# Calculate average agent latency per loop
+./query-logs.sh --task-id "$TASK_ID" --event-type agent_complete --format json | \
+  jq -r '.[] | "\(.loop) \(.details | fromjson | .latency_ms)"' | \
+  awk '{sum[$1]+=$2; count[$1]++} END {for (loop in sum) print loop, sum[loop]/count[loop]}'
+
+# Find parameter errors (for troubleshooting implementations)
+./query-logs.sh --task-id "$TASK_ID" --event-type parameter_error --format json
+
+# Track decision history
+./query-logs.sh --task-id "$TASK_ID" --event-type po_decision --format json | \
+  jq -r '.[] | "\(.iteration): \(.details | fromjson | .decision) - \(.details | fromjson | .reasoning)"'
+```
+
+## Logged Events in Orchestrator
+
+The orchestrator automatically logs:
+
+1. **Line ~643:** Swarm initialization with all configuration
+2. **Line ~811:** Each Loop 3 agent spawn with timeout
+3. **Line ~892:** Each Loop 3 agent completion with confidence and files changed
+4. **Line ~917:** Each Loop 3 agent failure with error details
+5. **Line ~1082:** Gate check failures with consensus scores
+6. **Line ~1115:** Gate check successes
+7. **Line ~1440:** Product Owner decisions with reasoning
+
+All logs include `2>/dev/null || true` to ensure logging failures don't break orchestration.
+
+## Performance
+
+- **Write latency:** ~5-10ms per log entry
+- **Query latency:** ~10-50ms for typical queries (< 1000 events)
+- **Storage:** ~500 bytes per event (compressed SQLite)
+- **Indexes:** Optimized for task_id, event_type, timestamp, level queries
+
+## Debugging
+
+### Check if logging is working
+
+```bash
+# Check database exists
+ls -lh data/cfn-loop.db
+
+# Count total log entries
+sqlite3 data/cfn-loop.db "SELECT COUNT(*) FROM cfn_loop_logs;"
+
+# Get latest 5 events
+./query-logs.sh --task-id "YOUR_TASK_ID" --limit 5 --format table
+```
+
+### Common issues
+
+**Issue:** "Error: Database not found"
+- **Cause:** No logs written yet
+- **Fix:** Run a CFN loop task to generate logs
+
+**Issue:** "Error: --details must be valid JSON"
+- **Cause:** Malformed JSON in details parameter
+- **Fix:** Validate JSON with `echo "$DETAILS" | jq empty`
+
+**Issue:** Logging fails silently
+- **Cause:** `2>/dev/null || true` suppresses errors
+- **Fix:** Remove `2>/dev/null` temporarily to see error messages
+
+## Web Portal Integration
+
+The web portal can query logs for real-time visibility:
+
+```typescript
+// Example: Fetch logs for task
+const logs = await fetch('/api/logs?task_id=cfn-task-123&event_type=agent_spawn');
+const events = await logs.json();
+
+// Display in timeline
+events.forEach(event => {
+  console.log(`${event.timestamp} [${event.loop}] ${event.agent_id}: ${event.event_type}`);
+});
+```
+
+## Retention
+
+- **Default:** Logs persist indefinitely in SQLite
+- **Recommended:** Implement cleanup job to delete logs older than 30 days for non-critical tasks
+- **Critical tasks:** Retain logs for audit trail
+
+```bash
+# Example: Delete logs older than 30 days
+sqlite3 data/cfn-loop.db "DELETE FROM cfn_loop_logs WHERE timestamp < datetime('now', '-30 days');"
+```
+
+## Privacy & Security
+
+- **Sensitive data:** Avoid logging secrets, API keys, or PII in details field
+- **Access control:** Database file permissions (chmod 600) restrict access
+- **Audit trail:** Logs include full decision reasoning for compliance
+
+## Future Enhancements
+
+- [ ] Structured logging levels (DEBUG for verbose agent output)
+- [ ] Log streaming to external systems (Elasticsearch, CloudWatch)
+- [ ] Automatic anomaly detection (high failure rates, long latencies)
+- [ ] Log rotation and archival
+- [ ] Web UI for log browsing and search

package/.claude/skills/redis-coordination/README.md

@@ -13,52 +13,53 @@
 2. Configure Redis connection in `config.json`
 3. Make scripts executable:
 ```bash
-chmod +x invoke-redis-pattern.sh test-waiting-mode.sh
+chmod +x invoke-waiting-mode.sh
 ```
 
 ### Basic Usage
 
-#### Waiting Mode
+#### Consensus Collection (Updated)
 ```bash
-# Agent enters waiting mode
-./invoke-redis-pattern.sh wait \
+# Agent reports results
+./invoke-waiting-mode.sh report \
   --task-id "my-task" \
   --agent-id "agent-1" \
-  --context "iteration-1"
-
-# Coordinator wakes agent
-./invoke-redis-pattern.sh wake \
-  --task-id "my-task" \
-  --agent-id "agent-1" \
-  --payload '{"instruction": "proceed"}'
-```
-
-#### Consensus Collection
-```bash
-# Multiple agents report results
-./invoke-redis-pattern.sh report \
-  --task-id "my-task" \
-  --agent-id "agent-1" \
-  --confidence 0.95 \
-  --result '{"status": "completed"}'
+  --confidence 0.95
 
 # Collect and evaluate consensus
-./invoke-redis-pattern.sh collect \
+./invoke-waiting-mode.sh collect \
   --task-id "my-task" \
   --agent-ids "agent-1,agent-2,agent-3"
 ```
 
-## Testing
-Run comprehensive tests:
-```bash
-./test-waiting-mode.sh
-```
+## Important Changes in P7 (Redis Script Cleanup)
 
-## Configuration Options
-See `SKILL.md` for detailed configuration and usage instructions.
+### Deprecation Notices
+- 🚨 `enter` and `wake` subcommands are NO LONGER SUPPORTED
+- Agents should exit cleanly without waiting mode
+- Coordinator spawns agents directly
+- Fork-ID references have been removed
+
+### Migration Guide
+- Update agent scripts to exit cleanly after task
+- Remove manual waiting mode calls
+- Use direct agent spawning in orchestrator
+
+## Script Categories
+- **Production Scripts**:
+  - `invoke-waiting-mode.sh`: Redis coordination wrapper
+  - `orchestrate-cfn-loop.sh`: CFN Loop orchestration
+- **Demos and Tests**: Located in `./demos/`
 
 ## Performance
 - Zero-token waiting
 - Sub-100ms wake-up latency
 - Supports 10+ concurrent agents
 - Configurable consensus thresholds
+
+## Configuration Options
+See `SKILL.md` for detailed configuration and usage instructions.
+
+## Troubleshooting
+- If you encounter issues with old scripts, refer to migration guide
+- Test scripts are available in `./demos/` directory