@pennyfarthing/core 6.6.0

package/pennyfarthing-dist/guides/patterns/fan-out-fan-in-pattern.md

@@ -0,0 +1,574 @@
+# Fan-out/Fan-in Pattern
+
+**Pattern Type:** Parallel Execution with Aggregation
+**Agents Involved:** Orchestrating agent + multiple parallel workers
+**Coordination Mechanism:** Task tool with `run_in_background` + TaskOutput
+
+## Problem Statement
+
+Sequential agent workflows create bottlenecks when processing independent work items:
+
+1. **Time Waste:** Waiting for each agent to complete before starting the next, even when work is independent
+2. **Underutilization:** Compute resources sit idle while one agent runs
+3. **Scaling Limits:** Total time grows linearly with work item count
+4. **Unnecessary Coupling:** Independent operations blocked by unrelated predecessors
+
+Without parallel execution, workflows may:
+- Take N times longer for N independent items
+- Miss opportunities for concurrent data gathering
+- Create artificial dependencies between unrelated work
+- Limit throughput when coordinating across multiple repositories
+
+## Solution
+
+Use **parallel Task invocations** to fan work out to multiple agents, then **aggregate results** when they complete. This creates a scatter-gather architecture:
+
+```
+Orchestrator
+    │
+    ├───► Agent A ──┐
+    ├───► Agent B ──┼──► Collect Results ──► Continue
+    └───► Agent C ──┘
+```
+
+The key insight: **Independent work items can execute concurrently** using Claude Code's Task tool parallelism.
+
+### Two Parallelism Modes
+
+1. **Implicit Parallelism:** Multiple Task calls in a single message execute concurrently
+2. **Explicit Background:** Use `run_in_background: true` for long-running tasks with manual result collection
+
+## State Diagram
+
+```mermaid
+stateDiagram-v2
+    [*] --> ORCHESTRATE: Start parallel work
+
+    ORCHESTRATE --> FAN_OUT: Identify independent items
+
+    FAN_OUT --> PARALLEL: Spawn Task calls
+    note right of FAN_OUT: Multiple Task tools\nin single message
+
+    state PARALLEL {
+        Agent_A --> Collecting
+        Agent_B --> Collecting
+        Agent_C --> Collecting
+    }
+
+    Collecting --> AGGREGATE: All complete
+
+    AGGREGATE --> CONTINUE: Results merged
+    CONTINUE --> [*]: Workflow proceeds
+
+    state Timeout {
+        [*] --> PARTIAL_FAILURE
+        PARTIAL_FAILURE --> AGGREGATE: Handle missing
+    }
+
+    note right of PARALLEL: Concurrent execution
+    note right of AGGREGATE: Merge results
+```
+
+### ASCII Alternative
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                      FAN-OUT/FAN-IN PATTERN                             │
+└─────────────────────────────────────────────────────────────────────────┘
+
+    ┌─────────────┐
+    │ ORCHESTRATE │
+    │  (decide    │
+    │   work)     │
+    └──────┬──────┘
+           │
+           │ Identify N independent work items
+           │
+           ▼
+    ┌─────────────┐
+    │  FAN-OUT    │
+    │  (spawn)    │
+    └──────┬──────┘
+           │
+     ┌─────┼─────┬─────┐
+     │     │     │     │
+     ▼     ▼     ▼     ▼
+   ┌───┐ ┌───┐ ┌───┐ ┌───┐
+   │ A │ │ B │ │ C │ │ D │  ◄── Parallel agents
+   └─┬─┘ └─┬─┘ └─┬─┘ └─┬─┘
+     │     │     │     │
+     │     │     │     │      Work executes concurrently
+     ▼     ▼     ▼     ▼
+   ┌───┐ ┌───┐ ┌───┐ ┌───┐
+   │ ✓ │ │ ✓ │ │ ✗ │ │ ✓ │  ◄── Results (may have failures)
+   └─┬─┘ └─┬─┘ └─┬─┘ └─┬─┘
+     │     │     │     │
+     └─────┴─────┴─────┘
+               │
+               ▼
+    ┌─────────────────┐
+    │    FAN-IN       │
+    │   (collect)     │
+    └────────┬────────┘
+             │
+             │ Aggregate results, handle partial failures
+             │
+             ▼
+    ┌─────────────────┐
+    │    CONTINUE     │
+    │   (proceed)     │
+    └─────────────────┘
+```
+
+### Timing Comparison
+
+```
+Sequential (without pattern):
+┌──────────────────────────────────────────────────────────────────────┐
+│ Agent A │        │ Agent B │        │ Agent C │        │ Agent D │  │
+│  (5s)   │ wait   │  (3s)   │ wait   │  (4s)   │ wait   │  (2s)   │  │
+└──────────────────────────────────────────────────────────────────────┘
+Total: 14 seconds + wait overhead
+
+Parallel (with pattern):
+┌──────────────────────────────────────────────────────────────────────┐
+│ Agent A │████████████                                                │
+│ Agent B │██████                                                      │
+│ Agent C │████████                                                    │
+│ Agent D │████                                                        │
+└──────────────────────────────────────────────────────────────────────┘
+Total: 5 seconds (longest task)
+```
+
+## Implementation
+
+### Method 1: Implicit Parallelism (Multiple Task Calls)
+
+When you include multiple Task tool invocations in a single response, Claude Code executes them concurrently:
+
+```yaml
+# In a single agent response, call multiple Tasks
+# All three execute in parallel automatically
+
+Task:
+  subagent_type: "Explore"
+  description: "Check frontend repo"
+  prompt: |
+    Scan frontend/ directory for TypeScript errors.
+    Report count and severity.
+
+Task:
+  subagent_type: "Explore"
+  description: "Check backend repo"
+  prompt: |
+    Scan api/ directory for Go lint issues.
+    Report count and severity.
+
+Task:
+  subagent_type: "Explore"
+  description: "Check shared libs"
+  prompt: |
+    Scan libs/ directory for outdated dependencies.
+    Report count and severity.
+```
+
+Results return when all tasks complete. The agent then aggregates them.
+
+### Method 2: Explicit Background Execution
+
+For long-running tasks or when you need to do work while waiting:
+
+```yaml
+# Fan-out: Start background tasks
+Task:
+  subagent_type: "testing-runner"
+  description: "Run backend tests"
+  run_in_background: true
+  prompt: |
+    REPOS: api
+    Execute test suite and report results.
+
+Task:
+  subagent_type: "testing-runner"
+  description: "Run frontend tests"
+  run_in_background: true
+  prompt: |
+    REPOS: frontend
+    Execute test suite and report results.
+```
+
+This returns immediately with task IDs. To collect results:
+
+```yaml
+# Fan-in: Collect results (blocking)
+TaskOutput:
+  task_id: "abc123"  # ID from first Task result
+  block: true
+  timeout: 120000    # 2 minute timeout
+
+TaskOutput:
+  task_id: "def456"  # ID from second Task result
+  block: true
+  timeout: 120000
+```
+
+### Result Aggregation Pattern
+
+After collecting results, merge them into a unified structure:
+
+```markdown
+## Multi-Repo Status Report
+
+### Summary
+| Repo | Status | Issues | Time |
+|------|--------|--------|------|
+| frontend | PASS | 0 | 3.2s |
+| api | FAIL | 3 | 5.1s |
+| libs | PASS | 0 | 1.8s |
+
+### Total Issues: 3
+### Overall Status: FAILING
+
+### Details
+[Aggregated from individual reports...]
+```
+
+### Real Example: Parallel Repo Status Check
+
+**Use Case:** Check git status across multiple repositories simultaneously
+
+```yaml
+# Orchestrator spawns parallel status checks
+Task:
+  subagent_type: "workflow-status-check"
+  description: "Check pennyfarthing status"
+  prompt: |
+    REPO: pennyfarthing
+    PROJECT_ROOT: /Users/dev/Projects/pennyfarthing
+    Report git status, branch, and uncommitted changes.
+
+Task:
+  subagent_type: "workflow-status-check"
+  description: "Check API status"
+  prompt: |
+    REPO: api
+    PROJECT_ROOT: /Users/dev/Projects/api
+    Report git status, branch, and uncommitted changes.
+
+Task:
+  subagent_type: "workflow-status-check"
+  description: "Check UI status"
+  prompt: |
+    REPO: ui
+    PROJECT_ROOT: /Users/dev/Projects/ui
+    Report git status, branch, and uncommitted changes.
+```
+
+### Real Example: Parallel File Analysis
+
+**Use Case:** Analyze multiple files to build context for a story
+
+```yaml
+# SM agent gathers context from multiple files in parallel
+Task:
+  subagent_type: "sm-file-summary"
+  description: "Summarize service layer"
+  prompt: |
+    FILES: internal/service/*.go
+    Create condensed summary focusing on public interfaces.
+
+Task:
+  subagent_type: "sm-file-summary"
+  description: "Summarize handlers"
+  prompt: |
+    FILES: internal/handlers/*.go
+    Create condensed summary focusing on HTTP endpoints.
+
+Task:
+  subagent_type: "sm-file-summary"
+  description: "Summarize models"
+  prompt: |
+    FILES: internal/models/*.go
+    Create condensed summary focusing on data structures.
+```
+
+## When to Use
+
+### Ideal Scenarios
+
+| Scenario | Why Fan-out/Fan-in Works |
+|----------|--------------------------|
+| Multi-repo operations | Each repo is independent |
+| Gathering diverse data | File scans don't affect each other |
+| Running test suites | Tests across repos/packages run independently |
+| Status checks | Checking N things concurrently |
+| Independent file processing | Batch processing without dependencies |
+
+### Decision Heuristic
+
+```
+Is the work...
+├── Truly independent (no shared state)?
+│   └── YES → Fan-out is safe
+├── Using shared resources (same files, DB)?
+│   └── CAUTION → May need synchronization
+├── Order-dependent (A before B)?
+│   └── NO fan-out → Use sequential
+├── Aggregatable (results can be merged)?
+│   └── YES → Fan-in will work
+└── Failure-tolerant (partial success OK)?
+    └── YES → Fan-out with error handling
+```
+
+### Concurrency Guidelines
+
+| Work Items | Recommendation |
+|------------|----------------|
+| 2-5 | Implicit parallelism (multiple Task calls) |
+| 5-10 | Explicit background with batching |
+| 10+ | Consider chunking into batches |
+
+## Error Recovery
+
+### Partial Failure Handling
+
+When some parallel tasks fail while others succeed:
+
+```markdown
+## Aggregated Results
+
+### Successful (3/4)
+| Task | Status | Result |
+|------|--------|--------|
+| Frontend check | SUCCESS | Clean |
+| Backend check | SUCCESS | Clean |
+| Lib check | SUCCESS | Clean |
+
+### Failed (1/4)
+| Task | Status | Error |
+|------|--------|-------|
+| DB check | TIMEOUT | Connection refused |
+
+### Decision
+**Proceeding with partial results.** DB check can be retried independently.
+```
+
+### Timeout Strategy
+
+```yaml
+# Set appropriate timeouts based on expected work
+TaskOutput:
+  task_id: "task-123"
+  block: true
+  timeout: 30000   # 30s for quick tasks
+
+TaskOutput:
+  task_id: "task-456"
+  block: true
+  timeout: 300000  # 5min for test suites
+```
+
+### Retry Pattern
+
+```
+For each failed task:
+1. Log failure reason
+2. Determine if retryable (timeout vs. permanent error)
+3. Retry up to 2 times with backoff
+4. If still failing, mark as failed in aggregate
+5. Continue with successful results if acceptable
+```
+
+### Graceful Degradation
+
+When complete failure is unacceptable:
+
+```markdown
+## Fallback Strategy
+
+If parallel execution fails completely:
+1. Fall back to sequential execution
+2. Log degradation for monitoring
+3. Continue workflow with slower path
+```
+
+## Anti-Patterns
+
+### Dependent Tasks in Parallel
+
+**Wrong:**
+```yaml
+# These have a dependency - B needs A's output!
+Task:
+  subagent_type: "analyzer"
+  prompt: "Analyze the codebase and identify modules"
+
+Task:
+  subagent_type: "reporter"
+  prompt: "Report on the modules identified above"  # Can't access A's output!
+```
+
+**Correct:**
+```yaml
+# Sequential for dependent work
+Task:
+  subagent_type: "analyzer"
+  prompt: "Analyze the codebase and identify modules"
+# Wait for result, then:
+Task:
+  subagent_type: "reporter"
+  prompt: "Report on these modules: {modules_from_A}"
+```
+
+### Too Many Parallel Tasks
+
+**Wrong:**
+```yaml
+# Spawning 50 parallel tasks overwhelms resources
+for file in all_files:
+    Task:
+      prompt: "Process {file}"
+      run_in_background: true
+```
+
+**Correct:**
+```yaml
+# Batch into manageable chunks
+Task:
+  prompt: "Process files 1-10: {batch_1}"
+  run_in_background: true
+
+Task:
+  prompt: "Process files 11-20: {batch_2}"
+  run_in_background: true
+# ... continue with reasonable batch sizes
+```
+
+### Ignoring Failed Tasks
+
+**Wrong:**
+```markdown
+# Silently proceeding despite failures
+All tasks returned. Continuing with workflow...
+```
+
+**Correct:**
+```markdown
+# Explicit failure handling
+## Task Results
+- 3/4 tasks succeeded
+- 1/4 tasks failed (timeout on repo-c)
+
+## Decision
+Partial failure is acceptable for status check.
+Proceeding with available data.
+```
+
+### No Timeout on Background Tasks
+
+**Wrong:**
+```yaml
+TaskOutput:
+  task_id: "task-123"
+  block: true
+  # No timeout - could hang forever
+```
+
+**Correct:**
+```yaml
+TaskOutput:
+  task_id: "task-123"
+  block: true
+  timeout: 60000  # Always set appropriate timeout
+```
+
+### Mixing Parallel and Sequential Without Awareness
+
+**Wrong:**
+```yaml
+# A and B are parallel, but developer thinks B waits for A
+Task:
+  description: "Create branch"  # A
+  prompt: "Create feature branch"
+
+Task:
+  description: "Commit to branch"  # B - races with A!
+  prompt: "Commit initial files"
+```
+
+**Correct:**
+```yaml
+# Make dependency explicit
+Task:
+  description: "Create branch and commit"
+  prompt: |
+    1. Create feature branch
+    2. Commit initial files
+    # Sequential within single task
+```
+
+## Comparison with Related Patterns
+
+| Pattern | Use When | Key Difference |
+|---------|----------|----------------|
+| **TDD Flow** | Sequential handoffs needed | One agent at a time |
+| **Helper Delegation** | Strategic/tactical split | Opus delegates to Haiku |
+| **Fan-out/Fan-in** | Independent parallel work | Multiple agents simultaneously |
+| **Approval Gates** | Human decision points | Blocking on external input |
+
+### Pattern Combinations
+
+Fan-out/fan-in works well with other patterns:
+
+```
+TDD Flow with parallel repo checks:
+SM ──► [Parallel status checks] ──► TEA ──► Dev ──► Reviewer ──► SM
+
+Helper delegation with parallel subagents:
+Opus ──► [Parallel Haiku tasks] ──► Opus continues
+```
+
+## Implementation Checklist
+
+When implementing fan-out/fan-in:
+
+- [ ] Verify work items are truly independent
+- [ ] Choose implicit vs. explicit parallelism
+- [ ] Set appropriate timeouts for each task type
+- [ ] Plan result aggregation format
+- [ ] Handle partial failures gracefully
+- [ ] Consider batch size for many items
+- [ ] Test timeout and failure scenarios
+- [ ] Log parallel execution for debugging
+
+## Related Patterns
+
+- **TDD Flow Pattern** (`tdd-flow-pattern.md`): Sequential coordination that may use fan-out for status checks
+- **Helper Delegation Pattern** (`helper-delegation-pattern.md`): Can be combined with parallel subagent spawning
+- **Approval Gates Pattern** (`approval-gates-pattern.md`): May pause fan-out for human decision points
+
+## References
+
+### Claude Code Task Tool
+- `Task`: Spawns agents, supports `run_in_background` for async execution
+- `TaskOutput`: Collects results from background tasks
+- Multiple Task calls in single message execute concurrently
+
+### Agent Definitions
+- Orchestrator: `agents/orchestrator.md` - Multi-agent coordination
+- SM: `agents/sm.md` - Story workflow orchestration
+- Testing Runner: `agents/testing-runner.md` - Parallel test execution candidate
+
+### Current Usage Opportunities
+
+| Component | Current | Parallel Opportunity |
+|-----------|---------|---------------------|
+| SM file summaries | Sequential | Parallel file analysis |
+| Repo status checks | Sequential | Simultaneous repo scanning |
+| Testing runner | Sequential per-repo | Parallel test suites |
+| Multi-repo PR creation | Sequential | Parallel branch operations |
+
+---
+
+*Last verified: 2026-01-06*
+*Pennyfarthing v4.0.0*