claude-flow-novice 2.0.0 → 2.0.1

package/docs/archive/2025-10-10-architecture/other-designs/claude-code-task-tool-constraints.md

@@ -0,0 +1,401 @@
+# Claude Code Task Tool Constraints Analysis
+
+**Date:** 2025-10-10
+**Context:** Production Blocking Coordination Implementation
+**User Question:** "If a coordinator ends early, we can't launch a new coordinator or revive it until all agents are done, correct?"
+
+---
+
+## Task Tool Behavior (Evidence-Based)
+
+### 1. Agent Lifecycle Model
+
+**Source:** Claude Code documentation and testing
+
+```
+Claude Code Agent Loop:
+while (message.has_tool_calls()) {
+  execute_tools();
+  feed_results_back_to_model();
+}
+// No tool calls → agent stops → waits for user input
+```
+
+**Key Findings:**
+1. **Agents terminate when they stop making tool calls** (no explicit termination command)
+2. **Sub-agents are stateless** - new instances of Claude Code with isolated context
+3. **No communication between running agents** - each operates independently
+4. **Parent cannot send messages to running sub-agents** - fire-and-forget model
+5. **Exit hooks can block termination** (exit code 2 forces continuation)
+
+### 2. Task Tool Launch Behavior
+
+**What Happens When You Call Task Tool:**
+
+```javascript
+// Single message spawning 3 coordinators
+Task("mesh-coordinator", "Coordinator-A: Implement features 1-35", "mesh-coordinator")
+Task("mesh-coordinator", "Coordinator-B: Implement features 36-70", "mesh-coordinator")
+Task("mesh-coordinator", "Coordinator-C: Review all and send retries", "mesh-coordinator")
+```
+
+**Runtime Behavior:**
+1. ✅ All 3 agents spawn **in parallel**
+2. ✅ Each runs in **isolated context** (cannot see each other's conversations)
+3. ✅ Each terminates **independently** when it stops making tool calls
+4. ❌ Parent **cannot communicate** with running agents after spawn
+5. ❌ Parent **cannot revive** terminated agents
+6. ❌ Agents **cannot signal** parent to spawn replacements
+7. ⚠️ Parent must **wait for ALL agents** to complete before next action
+
+---
+
+## Critical Constraint: Cannot Respawn Until All Complete
+
+### User's Question: "If a coordinator ends early, we can't launch a new coordinator or revive it until all agents are done, correct?"
+
+**Answer:** ✅ **CORRECT** - This is a fundamental constraint of Claude Code's Task tool.
+
+### Why This Matters for Blocking Coordination
+
+**Scenario from Layer 3 Test:**
+```
+Time 0:00 → Spawn 3 coordinators (A, B, C) in single message
+Time 2:30 → Coordinator-A completes, returns result
+Time 5:32 → Coordinator-B completes, returns result
+Time 6:30 → Coordinator-C completes, returns result
+```
+
+**What We CANNOT Do:**
+- ❌ At 2:30, spawn a new Coordinator-A replacement while B/C still running
+- ❌ At 2:30, send message to Coordinator-A to continue working
+- ❌ At 2:30, check Coordinator-C's status and adjust A's behavior
+- ❌ At 5:32, spawn retry agents for A/B while C is still running
+
+**What We CAN Do:**
+- ✅ At 6:30 (after ALL complete), review results and spawn new agents
+- ✅ At 6:30, spawn retry coordinators in a new batch
+- ✅ At 6:30, analyze what went wrong and adjust strategy
+
+---
+
+## Implications for Production Blocking Coordination Plan
+
+### Gap Analysis: Design vs Reality
+
+Our **blocking-coordination-pattern.md** assumed:
+
+```javascript
+// DESIGN ASSUMPTION (INVALID):
+async function completeImplementationPhase(coordinatorId) {
+  await spawnAllCoders();
+
+  // Enter blocking loop - wait for Coordinator-C to signal
+  while (Date.now() - startTime < timeout) {
+    const signal = await redis.get(`coordination:signal:${coordinatorId}:complete`);
+    if (signal === 'true') {
+      return; // Exit when signaled
+    }
+
+    // Check retry queue
+    const retryQueue = await redis.lrange(`coordination:retry:${coordinatorId}`, 0, -1);
+    if (retryQueue.length > 0) {
+      await processRetryRequests(coordinatorId, retryQueue); // ❌ CANNOT DO THIS
+    }
+
+    await sleep(5000);
+  }
+}
+```
+
+**Problem:** This assumes Coordinator-A can:
+1. ✅ Enter a blocking loop (wait for signal) - **POSSIBLE with bash script**
+2. ❌ Spawn new sub-agents while in blocking loop - **IMPOSSIBLE** (agent would need to make Task tool calls while waiting)
+3. ❌ Process retry requests by spawning agents mid-wait - **IMPOSSIBLE**
+
+### Reality Check: What's Actually Possible
+
+**Option 1: True Blocking (Current Implementation)**
+```bash
+#!/bin/bash
+# blocking-loop-fixed.sh - Coordinator-A stays alive but CANNOT spawn agents
+
+while true; do
+  # Check for completion signal
+  COMPLETE=$(redis-cli get "coordination:signal:coordinator-a:complete")
+  if [ "$COMPLETE" = "true" ]; then
+    exit 0
+  fi
+
+  # Check retry queue
+  RETRY_COUNT=$(redis-cli llen "coordination:retry:coordinator-a")
+  if [ "$RETRY_COUNT" -gt 0 ]; then
+    # ❌ CANNOT spawn retry agents here - we're in bash, not Claude context
+    # ✅ CAN log the retry request for post-completion processing
+    echo "⚠️ Retry request detected: $(redis-cli rpop coordination:retry:coordinator-a)"
+  fi
+
+  sleep 5
+done
+```
+
+**What This Achieves:**
+- ✅ Coordinator-A stays alive until signaled
+- ✅ Coordinator-A can detect retry requests in Redis
+- ❌ Coordinator-A CANNOT spawn new agents to handle retries
+- ❌ Coordinator-A CANNOT communicate retry needs back to parent
+
+**Option 2: Delayed Retry (Practical Alternative)**
+```javascript
+// Parent coordinator workflow
+
+// Round 1: Initial implementation
+Task("mesh-coordinator", "Coordinator-A: Implement features 1-35 then WAIT", "mesh-coordinator")
+Task("mesh-coordinator", "Coordinator-B: Implement features 36-70 then WAIT", "mesh-coordinator")
+Task("mesh-coordinator", "Coordinator-C: Review all, write retry list to Redis", "mesh-coordinator")
+
+// Wait for ALL to complete...
+
+// Round 2: Check results and spawn retries if needed
+const retryListA = await redis.lrange("coordination:retry:coordinator-a", 0, -1);
+const retryListB = await redis.lrange("coordination:retry:coordinator-b", 0, -1);
+
+if (retryListA.length > 0) {
+  Task("mesh-coordinator", `Coordinator-A-Retry: Fix ${retryListA.join(', ')}`, "mesh-coordinator")
+}
+
+if (retryListB.length > 0) {
+  Task("mesh-coordinator", `Coordinator-A-Retry: Fix ${retryListB.join(', ')}`, "mesh-coordinator")
+}
+
+// Wait for retry round to complete...
+
+// Round 3: Re-review
+Task("mesh-coordinator", "Coordinator-C: Re-review fixes", "mesh-coordinator")
+```
+
+**What This Achieves:**
+- ✅ Coordinators finish implementation phase cleanly
+- ✅ Review coordinator identifies issues and writes to Redis
+- ✅ Parent can spawn NEW retry coordinators after all complete
+- ✅ Multi-round iteration is possible
+- ❌ No "live retry" during blocking wait
+- ❌ Requires parent to orchestrate rounds
+
+---
+
+## Architectural Implications
+
+### 1. Blocking Coordination Pattern Must Change
+
+**Original Design (INVALID):**
+- Coordinators enter WAITING state
+- Coordinators poll for retry requests
+- Coordinators spawn retry sub-agents dynamically
+- Coordinators exit when review coordinator signals complete
+
+**Revised Design (VALID):**
+- Coordinators enter WAITING state (blocking loop)
+- Coordinators poll for **completion signal only**
+- Coordinators log retry requests but **do not process them**
+- Coordinators exit when review coordinator signals complete
+- **Parent spawns retry coordinators in subsequent round**
+
+### 2. Production Plan Phases Affected
+
+**Phase 1 (Core Fixes) - ⚠️ NEEDS REVISION:**
+- **Sprint 1.1 (Signal ACK Protocol)**: ✅ Still valid - one-way signaling works
+- **Sprint 1.2 (Dead Coordinator Detection)**: ✅ Still valid - heartbeat monitoring works
+- **Sprint 1.3 (Redis Health Check)**: ✅ Still valid - connection monitoring works
+- **Sprint 1.4 (Extended Timeout Testing)**: ✅ Still valid - timeout enforcement works
+
+**Phase 2 (Integration) - ⚠️ MAJOR REVISION NEEDED:**
+- **Sprint 2.1 (Swarm State Manager)**: ⚠️ Needs adjustment - cannot revive mid-execution
+- **Sprint 2.2 (Agent Lifecycle Hooks)**: ⚠️ Needs adjustment - hooks can't spawn agents
+- **Sprint 2.3 (CFN Loop State Machine)**: ⚠️ CRITICAL - retry logic must be parent-orchestrated
+
+**Phase 3 (Production Hardening) - ⚠️ NEEDS REVISION:**
+- **Sprint 3.2 (Auto-Recovery)**: ❌ INVALID - cannot auto-spawn replacements during execution
+- **Sprint 3.4 (Chaos Testing)**: ⚠️ Test scenarios need adjustment for spawn constraints
+
+### 3. CFN Loop Integration Impact
+
+**Current CFN Loop Assumption:**
+```
+Loop 3 (Implementation):
+  - Spawn agents
+  - Agents self-correct and retry
+  - Exit when all ≥0.75 confidence
+
+Loop 2 (Validation):
+  - Spawn validators
+  - Validators check Loop 3 results
+  - If <0.90 consensus, send retry requests to Loop 3 agents
+  - Loop 3 agents process retries
+```
+
+**Reality with Task Tool Constraints:**
+```
+Loop 3 (Implementation) - Round 1:
+  - Spawn agents in single message
+  - Agents complete work
+  - Agents report confidence scores
+  - ❌ CANNOT retry if <0.75 while validators running
+
+Loop 2 (Validation):
+  - Spawn validators after Loop 3 complete
+  - Validators check results
+  - Validators write retry needs to Redis
+  - Validators report consensus score
+
+Loop 3 (Implementation) - Round 2 (if needed):
+  - Parent reads retry needs from Redis
+  - Parent spawns NEW agents for retries
+  - New agents fix issues
+  - New agents report confidence scores
+
+Loop 2 (Validation) - Round 2:
+  - Spawn validators again
+  - Check if consensus ≥0.90
+```
+
+**This means:**
+- ✅ CFN Loop works but requires **sequential rounds**, not live retries
+- ✅ Redis coordination still valuable for passing retry context between rounds
+- ⚠️ Max 10 iterations for Loop 3 means max 10 **rounds of spawning**, not 10 retries within same spawn
+- ⚠️ Blocking coordination provides **timing synchronization** but not **dynamic retry spawning**
+
+---
+
+## Recommendations
+
+### Immediate Action Items
+
+**1. Update Production Plan (Phase 2/3)**
+- ❌ Remove "auto-recovery" sprint (Sprint 3.2) - not possible with Task tool
+- ⚠️ Revise Sprint 2.3 (CFN Loop State Machine) to use round-based retry model
+- ⚠️ Update test scenarios to reflect sequential round constraints
+
+**2. Clarify Blocking Coordination Use Case**
+- ✅ Blocking coordination ensures timing synchronization (A/B don't exit before C finishes)
+- ✅ Blocking coordination enables retry **planning** (detect retry needs, write to Redis)
+- ❌ Blocking coordination does NOT enable live retry **execution** within same spawn batch
+
+**3. Document Round-Based Retry Pattern**
+```javascript
+// CFN Loop with Round-Based Retries
+
+async function executeCFNLoop(objective) {
+  let round = 1;
+  const maxRounds = 10;
+
+  while (round <= maxRounds) {
+    // Loop 3: Implementation round
+    const loop3Results = await spawnImplementationCoordinators(round);
+
+    // Check confidence gate
+    const allAboveThreshold = loop3Results.every(r => r.confidence >= 0.75);
+    if (!allAboveThreshold) {
+      console.log(`Loop 3 Round ${round}: Below threshold, retrying...`);
+      round++;
+      continue;
+    }
+
+    // Loop 2: Validation round
+    const loop2Results = await spawnValidationCoordinators(loop3Results);
+
+    // Check consensus
+    const consensus = calculateConsensus(loop2Results);
+    if (consensus >= 0.90) {
+      console.log(`✅ Consensus achieved: ${consensus}`);
+      return { success: true, rounds: round };
+    }
+
+    // Extract retry needs from Redis
+    const retryNeeds = await extractRetryNeeds();
+    if (retryNeeds.length === 0) {
+      console.log(`❌ No progress path found after ${round} rounds`);
+      return { success: false, rounds: round };
+    }
+
+    // Prepare next round with retry context
+    round++;
+  }
+
+  console.log(`❌ Max rounds (${maxRounds}) reached`);
+  return { success: false, rounds: maxRounds };
+}
+```
+
+**4. Update Agent Prompts**
+- Coordinator-A/B prompts should clarify: "You will NOT spawn retry agents. Write retry needs to Redis and wait for completion signal."
+- Coordinator-C prompt should clarify: "Write retry list to Redis. Parent will spawn retry coordinators in next round."
+
+---
+
+## Summary: Task Tool Constraints
+
+| Feature | Possible? | Implementation |
+|---------|-----------|----------------|
+| Spawn multiple agents in parallel | ✅ YES | Single message with multiple Task calls |
+| Agents run independently | ✅ YES | Isolated contexts, no inter-agent communication |
+| Agent enters blocking wait state | ✅ YES | Bash script with sleep loop |
+| Agent detects Redis signals during wait | ✅ YES | Bash script polls Redis |
+| Agent spawns NEW agents during wait | ❌ NO | Agent cannot make Task tool calls while in bash loop |
+| Parent sends message to running agent | ❌ NO | Fire-and-forget model |
+| Parent spawns replacement for terminated agent | ⚠️ ONLY AFTER ALL COMPLETE | Must wait for entire batch to finish |
+| Agent revives itself after termination | ❌ NO | Stateless - no persistence |
+| Multi-round retry coordination | ✅ YES | Parent orchestrates sequential rounds |
+| Live retry within single batch | ❌ NO | Requires spawning, which breaks blocking wait |
+
+---
+
+## Revised Epic Scope
+
+**In-Scope (Still Valid):**
+- Signal ACK protocol for completion confirmation
+- Dead coordinator detection via heartbeat monitoring
+- Redis health checks and connection resilience
+- Timeout enforcement for blocking coordinators
+- State persistence across rounds (Redis/SQLite)
+- CFN Loop state machine with round-based retries
+- Observability dashboard for round tracking
+- Multi-round retry coordination with parent orchestration
+
+**Out-of-Scope (Newly Added):**
+- Live retry spawning during blocking wait
+- Agent self-revival after termination
+- Mid-execution agent replacement
+- Dynamic agent spawning within blocking loop
+- Inter-agent direct communication during execution
+
+**Revised Risk Profile:**
+- **Was:** critical-high-risk (auto-recovery, live retries)
+- **Now:** public-facing-medium-risk (sequential rounds, simpler coordination)
+
+---
+
+## Conclusion
+
+**User's Question:** "If a coordinator ends early, we can't launch a new coordinator or revive it until all agents are done, correct?"
+
+**Answer:** ✅ **CORRECT**
+
+This is a **fundamental architectural constraint** of Claude Code's Task tool. Our production blocking coordination plan must be revised to work within these constraints:
+
+1. ✅ Blocking coordination for **timing synchronization** (valid use case)
+2. ✅ Round-based retry coordination with **parent orchestration** (valid pattern)
+3. ❌ Live retry spawning during blocking wait (invalid - not possible)
+4. ❌ Agent self-revival or mid-execution replacement (invalid - not possible)
+
+**Next Steps:**
+1. Revise production plan Phases 2-3 to remove auto-recovery and live retry features
+2. Update CFN Loop documentation to clarify round-based retry model
+3. Test round-based coordination pattern with hello-world example
+4. Update consensus validation criteria to account for round overhead
+
+**Impact on Timeline:**
+- Phase 1 (Core Fixes): No impact - all features still valid
+- Phase 2 (Integration): -1 week (simpler without auto-recovery)
+- Phase 3 (Production Hardening): -1 week (fewer features to test)
+- **Revised Total:** 2 weeks (down from 4 weeks)