the-grid-cc 1.7.13 → 1.7.14

package/docs/DAEMON_ARCHITECTURE.md

@@ -0,0 +1,780 @@
+# The Grid - Daemon Mode Architecture
+
+Technical design document for long-running autonomous execution in The Grid.
+
+---
+
+## Executive Summary
+
+Daemon Mode enables The Grid to execute complex, multi-hour tasks without requiring an active user session. Users can "fire and forget" large projects, check progress asynchronously, and receive notifications when work completes or requires attention.
+
+**Key Capabilities:**
+- Long-running autonomous execution (hours to days)
+- Process survival across terminal closures
+- Progress monitoring and status checks
+- Graceful pause, resume, and cancellation
+- Crash recovery and checkpoint-based resumption
+
+---
+
+## Current State Analysis
+
+### What The Grid Has Today
+
+| Feature | Current State |
+|---------|---------------|
+| **State Persistence** | `.grid/STATE.md` - survives terminal close |
+| **Checkpoint Protocol** | Programs return structured checkpoints |
+| **Warmth Transfer** | `lessons_learned` in SUMMARY.md |
+| **Scratchpad** | Live discovery sharing via `.grid/SCRATCHPAD.md` |
+| **Session Resume** | Manual via `/grid` checking STATE.md |
+
+### Current Limitations
+
+1. **Session-Bound Execution**: Work stops when user closes terminal
+2. **No Background Mode**: Can't run while user does other work
+3. **Manual Resume Required**: User must explicitly restart after pause
+4. **No Notifications**: No way to alert user when work completes
+5. **Context Window Bound**: Single-session context limits (200k tokens)
+
+---
+
+## Architecture Overview
+
+### Three-Layer Design
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                        DAEMON CONTROLLER                             │
+│  Lightweight process manager that survives terminal disconnection    │
+│  - Spawns/monitors Claude Code processes                            │
+│  - Manages checkpoint persistence                                    │
+│  - Handles notifications                                             │
+└─────────────────────────────────────────────────────────────────────┘
+                                ↓
+┌─────────────────────────────────────────────────────────────────────┐
+│                        SESSION ORCHESTRATOR                          │
+│  Master Control instance managing execution waves                    │
+│  - Coordinates parallel Programs                                     │
+│  - Handles inter-session state transfer                             │
+│  - Implements durable execution patterns                            │
+└─────────────────────────────────────────────────────────────────────┘
+                                ↓
+┌─────────────────────────────────────────────────────────────────────┐
+│                        WORKER PROGRAMS                               │
+│  Fresh Claude Code instances for actual work                         │
+│  - Planners, Executors, Recognizers, etc.                           │
+│  - Each gets fresh 200k context window                              │
+│  - Reports progress to Orchestrator                                 │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+### Daemon Controller (Process Layer)
+
+The Daemon Controller is a lightweight Node.js process that:
+
+1. **Spawns Claude Code sessions** via CLI
+2. **Monitors process health** with heartbeats
+3. **Persists state** to disk on every checkpoint
+4. **Survives disconnection** via `nohup` or similar
+5. **Sends notifications** via system notifications, webhooks, or email
+
+```javascript
+// Conceptual: daemon-controller.js
+class DaemonController {
+  constructor(config) {
+    this.stateFile = '.grid/daemon/state.json';
+    this.logFile = '.grid/daemon/daemon.log';
+    this.notificationHandler = new NotificationHandler(config);
+  }
+
+  async spawn(taskDescription) {
+    // 1. Create daemon state
+    const daemonId = generateId();
+    await this.persistState({ id: daemonId, status: 'starting' });
+
+    // 2. Spawn Claude Code in headless mode
+    const proc = spawn('claude', [
+      '--agent', 'grid-daemon-orchestrator',
+      '--input', taskDescription,
+      '--output', `.grid/daemon/${daemonId}/output.md`
+    ], { detached: true });
+
+    // 3. Detach from terminal
+    proc.unref();
+
+    return daemonId;
+  }
+
+  async checkStatus(daemonId) {
+    const state = await this.loadState(daemonId);
+    return {
+      status: state.status,
+      progress: state.progress,
+      lastCheckpoint: state.lastCheckpoint,
+      logs: await this.tailLogs(daemonId, 20)
+    };
+  }
+}
+```
+
+### Session Orchestrator (Coordination Layer)
+
+An enhanced Master Control that implements **durable execution**:
+
+```
+SESSION LIFECYCLE
+─────────────────
+
+1. INITIALIZE
+   ├── Load daemon state from disk
+   ├── Verify last checkpoint integrity
+   └── Determine resume point
+
+2. EXECUTE WAVE
+   ├── Spawn Programs (parallel within wave)
+   ├── Monitor via scratchpad polling
+   ├── Checkpoint after each Program completes
+   └── Persist full state to disk
+
+3. CHECKPOINT (after every wave)
+   ├── Serialize: conversation history, plan state, warmth
+   ├── Write to `.grid/daemon/{id}/checkpoint.json`
+   ├── Update progress in daemon state
+   └── Notify controller of progress
+
+4. HANDLE INTERRUPTION
+   ├── If graceful: complete current Program, checkpoint, exit
+   ├── If crash: controller detects via heartbeat timeout
+   └── Resume from last checkpoint on restart
+
+5. COMPLETE
+   ├── Final checkpoint with COMPLETE status
+   ├── Generate summary report
+   ├── Notify user
+   └── Clean up or archive daemon state
+```
+
+### State Persistence Model
+
+```yaml
+# .grid/daemon/{daemon-id}/checkpoint.json
+{
+  "version": "1.0",
+  "daemon_id": "20260123-143000-build-auth-api",
+  "created": "2026-01-23T14:30:00Z",
+  "updated": "2026-01-23T16:45:00Z",
+
+  "status": "executing",  # starting | executing | paused | checkpoint | complete | failed
+
+  "task": {
+    "description": "Build REST API with user authentication",
+    "mode": "autopilot",
+    "original_prompt": "..."
+  },
+
+  "progress": {
+    "current_wave": 2,
+    "total_waves": 4,
+    "completed_blocks": ["01", "02", "03"],
+    "current_block": "04",
+    "percent": 65
+  },
+
+  "execution_state": {
+    "plan_data": { /* Full Planner output */ },
+    "completed_summaries": { /* Block SUMMARY.md contents */ },
+    "warmth": { /* Accumulated lessons_learned */ },
+    "scratchpad_archive": [ /* All scratchpad entries */ ]
+  },
+
+  "checkpoint_stack": [
+    {
+      "type": "human-verify",
+      "block": "04",
+      "details": { /* Checkpoint data */ },
+      "created": "2026-01-23T16:45:00Z"
+    }
+  ],
+
+  "metrics": {
+    "start_time": "2026-01-23T14:30:00Z",
+    "elapsed_seconds": 8100,
+    "programs_spawned": 12,
+    "commits_made": 8,
+    "estimated_remaining_seconds": 4500
+  }
+}
+```
+
+---
+
+## Durable Execution Implementation
+
+### Checkpoint Protocol
+
+Based on research into durable execution patterns, checkpointing occurs at these boundaries:
+
+| Event | Checkpoint Contents | Recovery Action |
+|-------|---------------------|-----------------|
+| **Wave Complete** | Full state, all summaries | Resume next wave |
+| **Program Complete** | Program output, warmth | Resume wave |
+| **User Checkpoint** | Checkpoint data, pause reason | Wait for user |
+| **Crash** | Last known state | Verify + resume |
+| **Graceful Stop** | Full state + stop reason | Resume on restart |
+
+### Heartbeat & Health Monitoring
+
+```
+HEARTBEAT PROTOCOL
+──────────────────
+
+Orchestrator writes heartbeat every 30 seconds:
+  .grid/daemon/{id}/heartbeat.json
+  {
+    "timestamp": "2026-01-23T16:45:30Z",
+    "status": "executing",
+    "current_action": "Spawning executor-03"
+  }
+
+Controller considers Orchestrator dead if:
+  - No heartbeat for 2 minutes
+  - Process not found in system
+
+Recovery:
+  1. Controller reads last checkpoint
+  2. Spawns new Orchestrator with checkpoint
+  3. Orchestrator verifies git state
+  4. Resumes from checkpoint
+```
+
+### Crash Recovery
+
+```python
+def recover_from_crash(daemon_id):
+    """Recovery protocol after unexpected termination."""
+
+    # 1. Load last checkpoint
+    checkpoint = load_checkpoint(daemon_id)
+
+    # 2. Verify git state matches checkpoint
+    actual_commits = get_git_commits_since(checkpoint['task']['start_time'])
+    expected_commits = checkpoint['execution_state']['completed_summaries']
+
+    if commits_match(actual_commits, expected_commits):
+        # Clean recovery - resume from checkpoint
+        return spawn_orchestrator(checkpoint, mode='resume')
+    else:
+        # Dirty state - need reconciliation
+        return spawn_orchestrator(checkpoint, mode='reconcile')
+
+def reconcile_state(checkpoint, actual_git_state):
+    """Reconcile checkpoint with actual git state."""
+
+    # Find divergence point
+    last_matching_commit = find_last_matching(checkpoint, actual_git_state)
+
+    # Option A: Trust git, update checkpoint
+    # Option B: Trust checkpoint, revert git (dangerous)
+    # Default: Trust git, log discrepancy
+
+    updated_checkpoint = rebuild_from_git(last_matching_commit)
+    return updated_checkpoint
+```
+
+---
+
+## Multi-Session Context Management
+
+### The Context Window Problem
+
+A single Claude Code session is limited to ~200k tokens. Complex projects exceed this. Solution: **session chaining with warmth transfer**.
+
+```
+SESSION 1 (Waves 1-2)          SESSION 2 (Waves 3-4)
+┌─────────────────────┐        ┌─────────────────────┐
+│ Fresh 200k context  │        │ Fresh 200k context  │
+│                     │        │                     │
+│ - Execute Wave 1    │        │ - Load checkpoint   │
+│ - Execute Wave 2    │   →    │ - Apply warmth      │
+│ - Checkpoint        │        │ - Execute Wave 3    │
+│ - Extract warmth    │        │ - Execute Wave 4    │
+│ - Terminate         │        │ - Complete          │
+└─────────────────────┘        └─────────────────────┘
+         ↓                              ↑
+    checkpoint.json ────────────────────┘
+```
+
+### Session Handoff Protocol
+
+```python
+def handoff_to_new_session(current_checkpoint):
+    """Hand off to fresh session when context exhausted."""
+
+    # 1. Save current state
+    save_checkpoint(current_checkpoint)
+
+    # 2. Extract warmth (compressed learnings)
+    warmth = extract_warmth(current_checkpoint)
+
+    # 3. Terminate current session gracefully
+    terminate_session()
+
+    # 4. Spawn fresh session with minimal context
+    new_session = spawn_orchestrator({
+        'checkpoint_path': current_checkpoint.path,
+        'warmth': warmth,
+        'mode': 'continue'
+    })
+
+    return new_session
+```
+
+### Warmth Compression
+
+To fit learnings into new context windows, warmth is compressed:
+
+```yaml
+# Full warmth (too large for handoff)
+lessons_learned:
+  codebase_patterns:
+    - "Uses barrel exports in src/index.ts"
+    - "API routes in src/app/api/*/route.ts"
+    - "Uses Zod for validation everywhere"
+    - "Prisma client in src/lib/db.ts"
+    - ... (50 more patterns)
+
+# Compressed warmth (fits in context)
+warmth_compressed:
+  patterns: "barrel exports, Zod validation, Prisma in lib/db"
+  gotchas: "auth middleware runs first, timestamps UTC"
+  decisions: "chose JWT over sessions for statelessness"
+  critical_files: ["src/lib/auth.ts", "prisma/schema.prisma"]
+```
+
+---
+
+## User Interaction Patterns
+
+### Fire-and-Forget Launch
+
+```bash
+# User launches daemon
+/grid:daemon "Build complete e-commerce platform with Stripe integration"
+
+# Grid responds
+DAEMON SPAWNED
+══════════════
+
+ID: 20260123-143000-ecommerce
+Task: Build complete e-commerce platform with Stripe integration
+Mode: Autopilot
+
+Status: Planning phase
+Monitor: /grid:daemon status
+Stop: /grid:daemon stop
+
+You can close this terminal. Work continues in background.
+
+End of Line.
+```
+
+### Status Checking
+
+```bash
+# From any terminal
+/grid:daemon status
+
+# Output
+DAEMON STATUS
+═════════════
+
+ID: 20260123-143000-ecommerce
+Runtime: 2h 15m
+Status: Executing (Wave 3 of 5)
+
+Progress: [████████████░░░░░░░░] 60%
+
+Current: Block 07 - Payment Integration
+  ├─ Thread 7.1: Stripe SDK setup ✓
+  ├─ Thread 7.2: Checkout flow ⚡ In Progress
+  └─ Thread 7.3: Webhook handlers ○ Pending
+
+Recent Activity:
+  16:42 - executor-07: Implementing checkout session creation
+  16:38 - executor-06: Completed product catalog API
+  16:30 - recognizer: Verified Wave 2 artifacts ✓
+
+Commits: 14 made
+Est. Remaining: ~1h 30m
+
+End of Line.
+```
+
+### Checkpoints (User Attention Required)
+
+```bash
+# User gets notification (system notification, webhook, etc.)
+# "Grid Daemon needs your attention"
+
+/grid:daemon status
+
+# Output
+DAEMON CHECKPOINT
+═════════════════
+
+ID: 20260123-143000-ecommerce
+Status: AWAITING USER
+
+Checkpoint Type: human-verify
+Block: 08 - Stripe Webhooks
+
+What was built:
+- Stripe webhook endpoint at /api/webhooks/stripe
+- Event handlers for payment_intent.succeeded, .failed
+- Signature verification middleware
+
+How to verify:
+1. Run: stripe listen --forward-to localhost:3000/api/webhooks/stripe
+2. In another terminal: stripe trigger payment_intent.succeeded
+3. Check logs show "Payment succeeded" event processed
+
+Resume: /grid:daemon resume "approved"
+   Or: /grid:daemon resume "Issue: webhook not receiving events"
+
+End of Line.
+```
+
+### Resume After Checkpoint
+
+```bash
+/grid:daemon resume "approved"
+
+# Output
+DAEMON RESUMED
+══════════════
+
+Checkpoint cleared. Continuing execution...
+
+Current: Block 09 - Order Management
+Status: Executing
+
+End of Line.
+```
+
+---
+
+## Notification System
+
+### Notification Triggers
+
+| Event | Default Notification | Configurable |
+|-------|---------------------|--------------|
+| **Daemon Started** | Log only | Yes |
+| **Wave Complete** | None | Yes |
+| **Checkpoint Reached** | System notification | Yes |
+| **Error/Failure** | System notification + sound | Yes |
+| **Daemon Complete** | System notification | Yes |
+| **Stall Detected** | After 30min inactivity | Yes |
+
+### Notification Channels
+
+```yaml
+# .grid/config.json
+{
+  "daemon": {
+    "notifications": {
+      "system": true,           # macOS/Windows native notifications
+      "sound": true,            # Audio alert on checkpoint/complete
+      "webhook": null,          # POST to URL on events
+      "email": null,            # Email notifications (requires setup)
+      "slack": null             # Slack webhook URL
+    },
+    "notify_on": {
+      "checkpoint": true,
+      "complete": true,
+      "error": true,
+      "wave_complete": false,
+      "stall": true
+    },
+    "stall_threshold_minutes": 30
+  }
+}
+```
+
+### System Notification Implementation
+
+```javascript
+// Using node-notifier for cross-platform notifications
+const notifier = require('node-notifier');
+
+function notifyUser(event) {
+  notifier.notify({
+    title: 'The Grid',
+    message: formatEventMessage(event),
+    icon: path.join(__dirname, 'grid-icon.png'),
+    sound: event.type === 'checkpoint' || event.type === 'complete',
+    wait: event.type === 'checkpoint'  // Keep notification until dismissed
+  });
+}
+```
+
+---
+
+## Implementation Phases
+
+### Phase 1: Foundation (Current Claude Code Capabilities)
+
+**What's possible today:**
+
+1. **Manual daemon pattern** using `nohup claude ... &`
+2. **State persistence** via existing `.grid/STATE.md`
+3. **Checkpoint-based resume** via `/grid` reading STATE.md
+4. **Background agents** via Claude Code v2.0.60+ `Ctrl+B`
+
+**Implementation:**
+- Enhance STATE.md with daemon-specific fields
+- Create `/grid:daemon` command that sets up state and runs in background
+- Use Claude Code's native background agent support where available
+
+### Phase 2: Process Management (Requires External Tooling)
+
+**Needs:**
+- Daemon controller process (Node.js or shell script)
+- Process monitoring and heartbeat
+- Crash recovery automation
+
+**Implementation:**
+```bash
+# grid-daemon-launcher.sh
+#!/bin/bash
+DAEMON_ID=$(date +%Y%m%d-%H%M%S)-$(echo "$1" | tr ' ' '-' | head -c 20)
+DAEMON_DIR=".grid/daemon/$DAEMON_ID"
+mkdir -p "$DAEMON_DIR"
+
+# Save task
+echo "$1" > "$DAEMON_DIR/task.txt"
+
+# Launch Claude Code in background
+nohup claude --print --dangerouslySkipPermissions \
+  -p "$(cat ~/.claude/commands/grid/daemon-executor.md)" \
+  --input "$1" \
+  > "$DAEMON_DIR/output.log" 2>&1 &
+
+echo $! > "$DAEMON_DIR/pid"
+echo "Daemon $DAEMON_ID started"
+```
+
+### Phase 3: Full Daemon Mode (Requires Claude Code Changes)
+
+**Would need from Claude Code:**
+- Native daemon/service mode
+- IPC for status queries
+- Built-in notification system
+- Multi-session orchestration
+
+**Proposal for Claude Code team:**
+```
+Feature Request: Daemon Mode for Claude Code
+
+Use Case: Long-running autonomous development tasks
+
+Requested Capabilities:
+1. `claude daemon start "task"` - Launch headless session
+2. `claude daemon status <id>` - Query running daemon
+3. `claude daemon stop <id>` - Graceful termination
+4. `claude daemon list` - Show all running daemons
+5. Automatic checkpoint/resume on crash
+6. Native system notifications
+```
+
+---
+
+## Security Considerations
+
+### Sandboxing
+
+Daemon mode inherits Claude Code's permission model:
+- `--dangerouslySkipPermissions` should NOT be used in daemon mode
+- Each operation still requires appropriate permissions
+- File system access limited to project directory
+
+### Resource Limits
+
+```yaml
+# Daemon resource configuration
+daemon:
+  max_runtime_hours: 24          # Hard limit on execution time
+  max_programs_parallel: 5       # Limit concurrent Programs
+  max_commits_per_hour: 20       # Rate limit commits
+  max_file_modifications: 100    # Safety limit on file changes
+  require_approval_after: 50     # Force checkpoint after N commits
+```
+
+### Audit Trail
+
+All daemon activity logged to `.grid/daemon/{id}/audit.log`:
+```
+2026-01-23T14:30:00Z | START | Task: "Build e-commerce platform"
+2026-01-23T14:32:15Z | SPAWN | Planner (model: opus)
+2026-01-23T14:35:42Z | PLAN | 12 blocks, 5 waves
+2026-01-23T14:36:00Z | SPAWN | Executor-01 (block: 01)
+2026-01-23T14:42:18Z | COMMIT | abc123 "feat(01): Initialize project"
+...
+```
+
+---
+
+## Failure Modes & Mitigations
+
+| Failure Mode | Detection | Mitigation |
+|--------------|-----------|------------|
+| **Claude Code crash** | Heartbeat timeout | Auto-restart from checkpoint |
+| **System reboot** | Daemon controller starts on boot | Resume from checkpoint |
+| **Context exhaustion** | Token count monitoring | Session handoff |
+| **API rate limit** | 429 response | Exponential backoff |
+| **Git conflict** | Merge failure | Checkpoint, alert user |
+| **Infinite loop** | Stall detection | Alert user, pause |
+| **Permission denied** | Operation failure | Checkpoint, alert user |
+
+---
+
+## Metrics & Observability
+
+### Daemon Metrics
+
+```yaml
+# Exposed via /grid:daemon metrics
+metrics:
+  runtime_seconds: 8100
+  programs_spawned: 12
+  programs_failed: 0
+  commits_made: 8
+  files_created: 24
+  files_modified: 15
+  lines_written: 2847
+  checkpoints_hit: 3
+  checkpoint_wait_seconds: 120
+  context_resets: 1
+  warmth_transfers: 1
+```
+
+### Health Dashboard (Future)
+
+```
+DAEMON HEALTH
+═════════════
+
+Active Daemons: 2
+
+┌─────────────────────────────────────────────────────────────────────┐
+│ ID: ecommerce-build     Status: EXECUTING     Health: ●●●●○        │
+│ Runtime: 2h 15m         Progress: 60%         Est: 1h 30m          │
+├─────────────────────────────────────────────────────────────────────┤
+│ ID: api-refactor        Status: CHECKPOINT    Health: ●●●●●        │
+│ Runtime: 45m            Progress: 80%         Waiting: human-verify│
+└─────────────────────────────────────────────────────────────────────┘
+
+System Resources:
+  CPU: 12% (Claude Code processes)
+  Memory: 2.1 GB
+  Disk: 142 MB (.grid/ state)
+```
+
+---
+
+## Future Enhancements
+
+### Distributed Execution
+
+Multiple machines contributing to single daemon:
+- Shared state via cloud storage
+- Work distribution via queue
+- Merge reconciliation
+
+### Learning Across Daemons
+
+Global warmth database:
+- Patterns learned across all daemons
+- Shared gotchas and best practices
+- Per-project and global layers
+
+### Scheduled Daemons
+
+Cron-like scheduling:
+```bash
+/grid:daemon schedule "daily at 3am" "Run test suite and fix failures"
+```
+
+### Daemon Chaining
+
+Sequential daemon execution:
+```bash
+/grid:daemon chain \
+  "Build feature X" \
+  "Write tests for feature X" \
+  "Update documentation"
+```
+
+---
+
+## Appendix A: State File Schemas
+
+### checkpoint.json Schema
+
+```json
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "required": ["version", "daemon_id", "status", "task", "progress"],
+  "properties": {
+    "version": { "type": "string" },
+    "daemon_id": { "type": "string" },
+    "created": { "type": "string", "format": "date-time" },
+    "updated": { "type": "string", "format": "date-time" },
+    "status": {
+      "type": "string",
+      "enum": ["starting", "executing", "paused", "checkpoint", "complete", "failed"]
+    },
+    "task": {
+      "type": "object",
+      "properties": {
+        "description": { "type": "string" },
+        "mode": { "type": "string" },
+        "original_prompt": { "type": "string" }
+      }
+    },
+    "progress": {
+      "type": "object",
+      "properties": {
+        "current_wave": { "type": "integer" },
+        "total_waves": { "type": "integer" },
+        "completed_blocks": { "type": "array", "items": { "type": "string" } },
+        "current_block": { "type": "string" },
+        "percent": { "type": "integer" }
+      }
+    }
+  }
+}
+```
+
+---
+
+## Appendix B: Claude Code Feature Requests
+
+For full daemon mode capability, The Grid would benefit from these Claude Code enhancements:
+
+1. **Native daemon mode**: `claude daemon` subcommand
+2. **IPC channel**: Query running sessions without terminal
+3. **Notification API**: Hook into system notifications
+4. **Session serialization**: Export/import session state
+5. **Headless operation**: Run without TTY requirement
+6. **Multi-session coordination**: Built-in session chaining
+
+---
+
+*Document Version: 1.0*
+*Last Updated: 2026-01-23*
+*Author: Grid Program 1 (Daemon Architecture Specialist)*
+
+End of Line.