@pennyfarthing/core 7.4.1 → 7.5.0

package/pennyfarthing-dist/guides/patterns/fan-out-fan-in-pattern.md

@@ -1,574 +0,0 @@
-# Fan-out/Fan-in Pattern
-
-**Pattern Type:** Parallel Execution with Aggregation
-**Agents Involved:** Orchestrating agent + multiple parallel workers
-**Coordination Mechanism:** Task tool with `run_in_background` + TaskOutput
-
-## Problem Statement
-
-Sequential agent workflows create bottlenecks when processing independent work items:
-
-1. **Time Waste:** Waiting for each agent to complete before starting the next, even when work is independent
-2. **Underutilization:** Compute resources sit idle while one agent runs
-3. **Scaling Limits:** Total time grows linearly with work item count
-4. **Unnecessary Coupling:** Independent operations blocked by unrelated predecessors
-
-Without parallel execution, workflows may:
-- Take N times longer for N independent items
-- Miss opportunities for concurrent data gathering
-- Create artificial dependencies between unrelated work
-- Limit throughput when coordinating across multiple repositories
-
-## Solution
-
-Use **parallel Task invocations** to fan work out to multiple agents, then **aggregate results** when they complete. This creates a scatter-gather architecture:
-
-```
-Orchestrator
-    │
-    ├───► Agent A ──┐
-    ├───► Agent B ──┼──► Collect Results ──► Continue
-    └───► Agent C ──┘
-```
-
-The key insight: **Independent work items can execute concurrently** using Claude Code's Task tool parallelism.
-
-### Two Parallelism Modes
-
-1. **Implicit Parallelism:** Multiple Task calls in a single message execute concurrently
-2. **Explicit Background:** Use `run_in_background: true` for long-running tasks with manual result collection
-
-## State Diagram
-
-```mermaid
-stateDiagram-v2
-    [*] --> ORCHESTRATE: Start parallel work
-
-    ORCHESTRATE --> FAN_OUT: Identify independent items
-
-    FAN_OUT --> PARALLEL: Spawn Task calls
-    note right of FAN_OUT: Multiple Task tools\nin single message
-
-    state PARALLEL {
-        Agent_A --> Collecting
-        Agent_B --> Collecting
-        Agent_C --> Collecting
-    }
-
-    Collecting --> AGGREGATE: All complete
-
-    AGGREGATE --> CONTINUE: Results merged
-    CONTINUE --> [*]: Workflow proceeds
-
-    state Timeout {
-        [*] --> PARTIAL_FAILURE
-        PARTIAL_FAILURE --> AGGREGATE: Handle missing
-    }
-
-    note right of PARALLEL: Concurrent execution
-    note right of AGGREGATE: Merge results
-```
-
-### ASCII Alternative
-
-```
-┌─────────────────────────────────────────────────────────────────────────┐
-│                      FAN-OUT/FAN-IN PATTERN                             │
-└─────────────────────────────────────────────────────────────────────────┘
-
-    ┌─────────────┐
-    │ ORCHESTRATE │
-    │  (decide    │
-    │   work)     │
-    └──────┬──────┘
-           │
-           │ Identify N independent work items
-           │
-           ▼
-    ┌─────────────┐
-    │  FAN-OUT    │
-    │  (spawn)    │
-    └──────┬──────┘
-           │
-     ┌─────┼─────┬─────┐
-     │     │     │     │
-     ▼     ▼     ▼     ▼
-   ┌───┐ ┌───┐ ┌───┐ ┌───┐
-   │ A │ │ B │ │ C │ │ D │  ◄── Parallel agents
-   └─┬─┘ └─┬─┘ └─┬─┘ └─┬─┘
-     │     │     │     │
-     │     │     │     │      Work executes concurrently
-     ▼     ▼     ▼     ▼
-   ┌───┐ ┌───┐ ┌───┐ ┌───┐
-   │ ✓ │ │ ✓ │ │ ✗ │ │ ✓ │  ◄── Results (may have failures)
-   └─┬─┘ └─┬─┘ └─┬─┘ └─┬─┘
-     │     │     │     │
-     └─────┴─────┴─────┘
-               │
-               ▼
-    ┌─────────────────┐
-    │    FAN-IN       │
-    │   (collect)     │
-    └────────┬────────┘
-             │
-             │ Aggregate results, handle partial failures
-             │
-             ▼
-    ┌─────────────────┐
-    │    CONTINUE     │
-    │   (proceed)     │
-    └─────────────────┘
-```
-
-### Timing Comparison
-
-```
-Sequential (without pattern):
-┌──────────────────────────────────────────────────────────────────────┐
-│ Agent A │        │ Agent B │        │ Agent C │        │ Agent D │  │
-│  (5s)   │ wait   │  (3s)   │ wait   │  (4s)   │ wait   │  (2s)   │  │
-└──────────────────────────────────────────────────────────────────────┘
-Total: 14 seconds + wait overhead
-
-Parallel (with pattern):
-┌──────────────────────────────────────────────────────────────────────┐
-│ Agent A │████████████                                                │
-│ Agent B │██████                                                      │
-│ Agent C │████████                                                    │
-│ Agent D │████                                                        │
-└──────────────────────────────────────────────────────────────────────┘
-Total: 5 seconds (longest task)
-```
-
-## Implementation
-
-### Method 1: Implicit Parallelism (Multiple Task Calls)
-
-When you include multiple Task tool invocations in a single response, Claude Code executes them concurrently:
-
-```yaml
-# In a single agent response, call multiple Tasks
-# All three execute in parallel automatically
-
-Task:
-  subagent_type: "Explore"
-  description: "Check frontend repo"
-  prompt: |
-    Scan frontend/ directory for TypeScript errors.
-    Report count and severity.
-
-Task:
-  subagent_type: "Explore"
-  description: "Check backend repo"
-  prompt: |
-    Scan api/ directory for Go lint issues.
-    Report count and severity.
-
-Task:
-  subagent_type: "Explore"
-  description: "Check shared libs"
-  prompt: |
-    Scan libs/ directory for outdated dependencies.
-    Report count and severity.
-```
-
-Results return when all tasks complete. The agent then aggregates them.
-
-### Method 2: Explicit Background Execution
-
-For long-running tasks or when you need to do work while waiting:
-
-```yaml
-# Fan-out: Start background tasks
-Task:
-  subagent_type: "testing-runner"
-  description: "Run backend tests"
-  run_in_background: true
-  prompt: |
-    REPOS: api
-    Execute test suite and report results.
-
-Task:
-  subagent_type: "testing-runner"
-  description: "Run frontend tests"
-  run_in_background: true
-  prompt: |
-    REPOS: frontend
-    Execute test suite and report results.
-```
-
-This returns immediately with task IDs. To collect results:
-
-```yaml
-# Fan-in: Collect results (blocking)
-TaskOutput:
-  task_id: "abc123"  # ID from first Task result
-  block: true
-  timeout: 120000    # 2 minute timeout
-
-TaskOutput:
-  task_id: "def456"  # ID from second Task result
-  block: true
-  timeout: 120000
-```
-
-### Result Aggregation Pattern
-
-After collecting results, merge them into a unified structure:
-
-```markdown
-## Multi-Repo Status Report
-
-### Summary
-| Repo | Status | Issues | Time |
-|------|--------|--------|------|
-| frontend | PASS | 0 | 3.2s |
-| api | FAIL | 3 | 5.1s |
-| libs | PASS | 0 | 1.8s |
-
-### Total Issues: 3
-### Overall Status: FAILING
-
-### Details
-[Aggregated from individual reports...]
-```
-
-### Real Example: Parallel Repo Status Check
-
-**Use Case:** Check git status across multiple repositories simultaneously
-
-```yaml
-# Orchestrator spawns parallel status checks
-Task:
-  subagent_type: "workflow-status-check"
-  description: "Check pennyfarthing status"
-  prompt: |
-    REPO: pennyfarthing
-    PROJECT_ROOT: /Users/dev/Projects/pennyfarthing
-    Report git status, branch, and uncommitted changes.
-
-Task:
-  subagent_type: "workflow-status-check"
-  description: "Check API status"
-  prompt: |
-    REPO: api
-    PROJECT_ROOT: /Users/dev/Projects/api
-    Report git status, branch, and uncommitted changes.
-
-Task:
-  subagent_type: "workflow-status-check"
-  description: "Check UI status"
-  prompt: |
-    REPO: ui
-    PROJECT_ROOT: /Users/dev/Projects/ui
-    Report git status, branch, and uncommitted changes.
-```
-
-### Real Example: Parallel File Analysis
-
-**Use Case:** Analyze multiple files to build context for a story
-
-```yaml
-# SM agent gathers context from multiple files in parallel
-Task:
-  subagent_type: "sm-file-summary"
-  description: "Summarize service layer"
-  prompt: |
-    FILES: internal/service/*.go
-    Create condensed summary focusing on public interfaces.
-
-Task:
-  subagent_type: "sm-file-summary"
-  description: "Summarize handlers"
-  prompt: |
-    FILES: internal/handlers/*.go
-    Create condensed summary focusing on HTTP endpoints.
-
-Task:
-  subagent_type: "sm-file-summary"
-  description: "Summarize models"
-  prompt: |
-    FILES: internal/models/*.go
-    Create condensed summary focusing on data structures.
-```
-
-## When to Use
-
-### Ideal Scenarios
-
-| Scenario | Why Fan-out/Fan-in Works |
-|----------|--------------------------|
-| Multi-repo operations | Each repo is independent |
-| Gathering diverse data | File scans don't affect each other |
-| Running test suites | Tests across repos/packages run independently |
-| Status checks | Checking N things concurrently |
-| Independent file processing | Batch processing without dependencies |
-
-### Decision Heuristic
-
-```
-Is the work...
-├── Truly independent (no shared state)?
-│   └── YES → Fan-out is safe
-├── Using shared resources (same files, DB)?
-│   └── CAUTION → May need synchronization
-├── Order-dependent (A before B)?
-│   └── NO fan-out → Use sequential
-├── Aggregatable (results can be merged)?
-│   └── YES → Fan-in will work
-└── Failure-tolerant (partial success OK)?
-    └── YES → Fan-out with error handling
-```
-
-### Concurrency Guidelines
-
-| Work Items | Recommendation |
-|------------|----------------|
-| 2-5 | Implicit parallelism (multiple Task calls) |
-| 5-10 | Explicit background with batching |
-| 10+ | Consider chunking into batches |
-
-## Error Recovery
-
-### Partial Failure Handling
-
-When some parallel tasks fail while others succeed:
-
-```markdown
-## Aggregated Results
-
-### Successful (3/4)
-| Task | Status | Result |
-|------|--------|--------|
-| Frontend check | SUCCESS | Clean |
-| Backend check | SUCCESS | Clean |
-| Lib check | SUCCESS | Clean |
-
-### Failed (1/4)
-| Task | Status | Error |
-|------|--------|-------|
-| DB check | TIMEOUT | Connection refused |
-
-### Decision
-**Proceeding with partial results.** DB check can be retried independently.
-```
-
-### Timeout Strategy
-
-```yaml
-# Set appropriate timeouts based on expected work
-TaskOutput:
-  task_id: "task-123"
-  block: true
-  timeout: 30000   # 30s for quick tasks
-
-TaskOutput:
-  task_id: "task-456"
-  block: true
-  timeout: 300000  # 5min for test suites
-```
-
-### Retry Pattern
-
-```
-For each failed task:
-1. Log failure reason
-2. Determine if retryable (timeout vs. permanent error)
-3. Retry up to 2 times with backoff
-4. If still failing, mark as failed in aggregate
-5. Continue with successful results if acceptable
-```
-
-### Graceful Degradation
-
-When complete failure is unacceptable:
-
-```markdown
-## Fallback Strategy
-
-If parallel execution fails completely:
-1. Fall back to sequential execution
-2. Log degradation for monitoring
-3. Continue workflow with slower path
-```
-
-## Anti-Patterns
-
-### Dependent Tasks in Parallel
-
-**Wrong:**
-```yaml
-# These have a dependency - B needs A's output!
-Task:
-  subagent_type: "analyzer"
-  prompt: "Analyze the codebase and identify modules"
-
-Task:
-  subagent_type: "reporter"
-  prompt: "Report on the modules identified above"  # Can't access A's output!
-```
-
-**Correct:**
-```yaml
-# Sequential for dependent work
-Task:
-  subagent_type: "analyzer"
-  prompt: "Analyze the codebase and identify modules"
-# Wait for result, then:
-Task:
-  subagent_type: "reporter"
-  prompt: "Report on these modules: {modules_from_A}"
-```
-
-### Too Many Parallel Tasks
-
-**Wrong:**
-```yaml
-# Spawning 50 parallel tasks overwhelms resources
-for file in all_files:
-    Task:
-      prompt: "Process {file}"
-      run_in_background: true
-```
-
-**Correct:**
-```yaml
-# Batch into manageable chunks
-Task:
-  prompt: "Process files 1-10: {batch_1}"
-  run_in_background: true
-
-Task:
-  prompt: "Process files 11-20: {batch_2}"
-  run_in_background: true
-# ... continue with reasonable batch sizes
-```
-
-### Ignoring Failed Tasks
-
-**Wrong:**
-```markdown
-# Silently proceeding despite failures
-All tasks returned. Continuing with workflow...
-```
-
-**Correct:**
-```markdown
-# Explicit failure handling
-## Task Results
-- 3/4 tasks succeeded
-- 1/4 tasks failed (timeout on repo-c)
-
-## Decision
-Partial failure is acceptable for status check.
-Proceeding with available data.
-```
-
-### No Timeout on Background Tasks
-
-**Wrong:**
-```yaml
-TaskOutput:
-  task_id: "task-123"
-  block: true
-  # No timeout - could hang forever
-```
-
-**Correct:**
-```yaml
-TaskOutput:
-  task_id: "task-123"
-  block: true
-  timeout: 60000  # Always set appropriate timeout
-```
-
-### Mixing Parallel and Sequential Without Awareness
-
-**Wrong:**
-```yaml
-# A and B are parallel, but developer thinks B waits for A
-Task:
-  description: "Create branch"  # A
-  prompt: "Create feature branch"
-
-Task:
-  description: "Commit to branch"  # B - races with A!
-  prompt: "Commit initial files"
-```
-
-**Correct:**
-```yaml
-# Make dependency explicit
-Task:
-  description: "Create branch and commit"
-  prompt: |
-    1. Create feature branch
-    2. Commit initial files
-    # Sequential within single task
-```
-
-## Comparison with Related Patterns
-
-| Pattern | Use When | Key Difference |
-|---------|----------|----------------|
-| **TDD Flow** | Sequential handoffs needed | One agent at a time |
-| **Helper Delegation** | Strategic/tactical split | Opus delegates to Haiku |
-| **Fan-out/Fan-in** | Independent parallel work | Multiple agents simultaneously |
-| **Approval Gates** | Human decision points | Blocking on external input |
-
-### Pattern Combinations
-
-Fan-out/fan-in works well with other patterns:
-
-```
-TDD Flow with parallel repo checks:
-SM ──► [Parallel status checks] ──► TEA ──► Dev ──► Reviewer ──► SM
-
-Helper delegation with parallel subagents:
-Opus ──► [Parallel Haiku tasks] ──► Opus continues
-```
-
-## Implementation Checklist
-
-When implementing fan-out/fan-in:
-
-- [ ] Verify work items are truly independent
-- [ ] Choose implicit vs. explicit parallelism
-- [ ] Set appropriate timeouts for each task type
-- [ ] Plan result aggregation format
-- [ ] Handle partial failures gracefully
-- [ ] Consider batch size for many items
-- [ ] Test timeout and failure scenarios
-- [ ] Log parallel execution for debugging
-
-## Related Patterns
-
-- **TDD Flow Pattern** (`tdd-flow-pattern.md`): Sequential coordination that may use fan-out for status checks
-- **Helper Delegation Pattern** (`helper-delegation-pattern.md`): Can be combined with parallel subagent spawning
-- **Approval Gates Pattern** (`approval-gates-pattern.md`): May pause fan-out for human decision points
-
-## References
-
-### Claude Code Task Tool
-- `Task`: Spawns agents, supports `run_in_background` for async execution
-- `TaskOutput`: Collects results from background tasks
-- Multiple Task calls in single message execute concurrently
-
-### Agent Definitions
-- Orchestrator: `agents/orchestrator.md` - Multi-agent coordination
-- SM: `agents/sm.md` - Story workflow orchestration
-- Testing Runner: `agents/testing-runner.md` - Parallel test execution candidate
-
-### Current Usage Opportunities
-
-| Component | Current | Parallel Opportunity |
-|-----------|---------|---------------------|
-| SM file summaries | Sequential | Parallel file analysis |
-| Repo status checks | Sequential | Simultaneous repo scanning |
-| Testing runner | Sequential per-repo | Parallel test suites |
-| Multi-repo PR creation | Sequential | Parallel branch operations |
-
----
-
-*Last verified: 2026-01-06*
-*Pennyfarthing v4.0.0*