anvil-dev-framework 0.1.6

package/docs/specifications/CCS-RALPH-INTEGRATION-DESIGN.md

@@ -0,0 +1,633 @@
+# CCS-Ralph Integration Design
+
+**Date**: 2026-01-07
+**Author**: Claude (Ultrathink Analysis)
+**Related**: SPEC-ANV-context-checkpoints.md, Ralph Wiggum System
+
+---
+
+## Executive Summary
+
+Ralph Wiggum is a continuous-loop autonomous execution system. It WILL hit context limits frequently during overnight runs. The Context Checkpoint System (CCS) must be deeply integrated into Ralph to enable graceful handoffs within and across iterations.
+
+**Key Insight**: CCS must operate at TWO levels:
+1. **Micro-level**: Within a single Ralph iteration (single Claude Code session)
+2. **Macro-level**: Across Ralph iterations (loop continuity)
+
+---
+
+## Ralph Architecture Analysis
+
+### Current Components
+
+| Component | File | Purpose |
+|-----------|------|---------|
+| Command | `global/commands/ralph.md` | User-facing `/ralph` commands |
+| State Manager | `global/lib/ralph_state.py` | Python state management |
+| Loop Script | `global/tools/ralph-loop.sh` | External bash loop |
+| Stop Hook | `global/hooks/ralph_stop.sh` | Exit decision logic |
+
+### Current Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                     CURRENT RALPH FLOW                               │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  ralph-loop.sh                                                      │
+│  ┌──────────────────────────────────────────────────────────────┐   │
+│  │ while true; do                                                │   │
+│  │   cat PROMPT.md | npx claude-code --print                     │   │
+│  │   # Claude Code runs until session ends                       │   │
+│  │   # ralph_stop.sh fires as Stop hook                          │   │
+│  │   # Exit code decides: 0 = stop loop, 1 = continue            │   │
+│  │ done                                                          │   │
+│  └──────────────────────────────────────────────────────────────┘   │
+│                                                                     │
+│  ralph_stop.sh checks:                                              │
+│  ├── completion promise? → exit 0 (task done)                       │
+│  ├── fatal error? → exit 0 (unrecoverable)                          │
+│  ├── max iterations? → exit 0 (limit reached)                       │
+│  ├── circuit breaker? → exit 0 (stuck)                              │
+│  └── else → exit 1 (continue loop)                                  │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+### Critical Gap: No Context Awareness
+
+**The Problem**: Ralph has NO awareness of context window usage.
+
+| Scenario | Current Behavior | Consequence |
+|----------|------------------|-------------|
+| Context fills up | Auto-compact triggers | Lossy summarization, forgets context |
+| Auto-compact disabled | Session ends abruptly | No continuation, work lost |
+| Long iteration | Context exhausted mid-task | Incomplete work, no handoff |
+| Complex task | Multiple iterations needed | No context carryover |
+
+---
+
+## CCS-Ralph Integration Design
+
+### Two-Level Integration Model
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                  CCS-RALPH INTEGRATION MODEL                         │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  MACRO LEVEL: Across Iterations                                     │
+│  ┌──────────────────────────────────────────────────────────────┐   │
+│  │                     ralph-loop.sh                             │   │
+│  │  ┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐        │   │
+│  │  │ Iter 1 │───▶│ Iter 2 │───▶│ Iter 3 │───▶│ Iter N │        │   │
+│  │  └────────┘    └────────┘    └────────┘    └────────┘        │   │
+│  │       │              │              │              │          │   │
+│  │       ▼              ▼              ▼              ▼          │   │
+│  │  [Checkpoint]   [Resume]      [Resume]      [Complete]        │   │
+│  │                                                               │   │
+│  │  State passed via ralph-state.json + handoff docs             │   │
+│  └──────────────────────────────────────────────────────────────┘   │
+│                                                                     │
+│  MICRO LEVEL: Within Single Iteration                               │
+│  ┌──────────────────────────────────────────────────────────────┐   │
+│  │                Claude Code Session (Iteration N)              │   │
+│  │                                                               │   │
+│  │  ┌─────┐    ┌─────┐    ┌─────┐    ┌─────┐    ┌─────┐         │   │
+│  │  │Tool │───▶│Tool │───▶│Tool │───▶│Tool │───▶│Tool │         │   │
+│  │  │ #1  │    │ #2  │    │ #3  │    │ #N  │    │ #M  │         │   │
+│  │  └─────┘    └─────┘    └─────┘    └─────┘    └─────┘         │   │
+│  │     │          │          │          │          │             │   │
+│  │     ▼          ▼          ▼          ▼          ▼             │   │
+│  │   [20%]      [35%]      [55%]      [72%]      [86%]          │   │
+│  │                                       │          │             │   │
+│  │                                    L1 ⚠️      L2 🔶           │   │
+│  │                                  Warning    Checkpoint         │   │
+│  │                                               │                │   │
+│  │                                               ▼                │   │
+│  │                                    [Generate Handoff]          │   │
+│  │                                    [Update State]              │   │
+│  │                                    [Signal Stop]               │   │
+│  └──────────────────────────────────────────────────────────────┘   │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+### New Data Flow
+
+```
+Iteration N starts
+│
+├── 1. PROMPT.md loaded (includes CCS checkpoint data if resuming)
+│
+├── 2. PostToolUse hook fires after each tool call
+│   │
+│   ├── Poll context percentage from Claude Code
+│   │
+│   ├── L0-L1 (0-84%): Continue working normally
+│   │
+│   └── L2 (85%+): Trigger CCS checkpoint sequence
+│       │
+│       ├── A. Complete current edit (max 60 seconds)
+│       ├── B. Commit WIP with checkpoint message
+│       ├── C. Generate CCS handoff document
+│       ├── D. Update ralph-state.json with checkpoint data
+│       └── E. Signal session stop → goes to ralph_stop.sh
+│
+├── 3. ralph_stop.sh fires
+│   │
+│   ├── Check: Completion promise? → status=completed, exit 0
+│   │
+│   ├── Check: CCS checkpoint? → NEW PATH
+│   │   ├── Preserve checkpoint data
+│   │   ├── Regenerate PROMPT.md with resume context
+│   │   └── exit 1 (restart with checkpoint)
+│   │
+│   ├── Check: Fatal error? → status=fatal_error, exit 0
+│   ├── Check: Circuit breaker? → status=circuit_breaker, exit 0
+│   └── Default: exit 1 (continue)
+│
+└── 4. If exit 1: Loop continues with Iteration N+1
+    │
+    └── PROMPT.md now contains:
+        ├── Original task context
+        ├── CCS checkpoint summary
+        ├── Handoff document reference
+        └── Resume instructions
+```
+
+---
+
+## Enhanced State Model
+
+### ralph-state.json Additions
+
+```json
+{
+  "task_name": "Migrate Jest to Vitest",
+  "objective": "Convert all test files from Jest to Vitest",
+  "iteration": 5,
+  "started_at": "2026-01-07T00:00:00Z",
+  "status": "running",
+
+  // Existing fields...
+  "no_change_count": 0,
+  "last_diff_hash": "abc123",
+  "error_hashes": [],
+  "max_iterations": 50,
+  "completion_promise": "COMPLETE",
+  "todo_items": ["Item 3", "Item 4"],
+  "completed_items": ["Item 1", "Item 2"],
+
+  // NEW: CCS Integration Fields
+  "context_checkpoint": {
+    "active": true,
+    "level": "L2",
+    "percent_at_checkpoint": 87,
+    "timestamp": "2026-01-07T11:45:00Z",
+    "handoff_file": ".claude/handoffs/2026-01-07-1145.md",
+    "resume_summary": "Implementing OAuth callback handler, completed token validation",
+    "files_in_progress": [
+      {
+        "path": "src/auth/oauth.ts",
+        "lines": "45-120",
+        "description": "OAuth callback handler"
+      }
+    ],
+    "current_todo_item": "Implement OAuth callback",
+    "progress_on_item": "70%"
+  },
+
+  // NEW: Context history for pattern detection
+  "context_history": [
+    {"iteration": 1, "peak_percent": 45, "checkpoint": false},
+    {"iteration": 2, "peak_percent": 62, "checkpoint": false},
+    {"iteration": 3, "peak_percent": 88, "checkpoint": true},
+    {"iteration": 4, "peak_percent": 56, "checkpoint": false},
+    {"iteration": 5, "peak_percent": 87, "checkpoint": true}
+  ]
+}
+```
+
+### Enhanced PROMPT.md Template
+
+```markdown
+# Task: Migrate Jest to Vitest
+
+## Objective
+Convert all test files from Jest to Vitest
+
+## Current State
+- **Iteration**: 5
+- **Last action**: OAuth callback implementation (70% complete)
+- **Remaining items**: 3 of 5
+- **Started**: 2026-01-07T00:00:00Z
+
+---
+
+## CONTEXT CHECKPOINT RESUME
+
+> ⚠️ This iteration is resuming from a context checkpoint.
+> The previous session hit 87% context (L2 threshold).
+
+### What Was In Progress
+Implementing OAuth callback handler. Completed token validation,
+was working on callback URL parsing when checkpoint triggered.
+
+### Files To Resume
+| File | Lines | Status |
+|------|-------|--------|
+| src/auth/oauth.ts | 45-120 | 70% complete |
+
+### Handoff Document
+Read `.claude/handoffs/2026-01-07-1145.md` for full context.
+
+### Resume Instructions
+1. **DO NOT restart the task from scratch**
+2. Read the handoff document first
+3. Continue from line 90 of oauth.ts
+4. The token validation is complete, focus on URL parsing
+
+---
+
+## Instructions
+
+### Step 1: Check If Resuming
+If `context_checkpoint.active == true`:
+1. Read the handoff document
+2. Continue from where previous session left off
+3. DO NOT re-read files already understood
+
+### Step 2: Execute ONE Item
+... (rest of standard PROMPT.md template)
+```
+
+---
+
+## Hook Integration
+
+### PostToolUse Hook Enhancement
+
+Create new hook: `global/hooks/ralph_context_monitor.py`
+
+```python
+"""
+ralph_context_monitor.py - CCS Monitor for Ralph Mode
+
+Runs after each tool use during Ralph execution.
+Monitors context percentage and triggers checkpoints.
+"""
+
+import json
+import os
+import sys
+from pathlib import Path
+from datetime import datetime, timezone
+
+# Thresholds
+THRESHOLD_L1 = 70
+THRESHOLD_L2 = 85
+THRESHOLD_L3 = 95
+
+STATE_FILE = os.environ.get("RALPH_STATE_FILE", ".claude/ralph-state.json")
+
+
+def get_context_percent():
+    """Get current context percentage from Claude Code."""
+    # Parse from environment or Claude Code API
+    # This would be provided by Claude Code's context tracking
+    return int(os.environ.get("CLAUDE_CONTEXT_PERCENT", "50"))
+
+
+def is_ralph_active():
+    """Check if Ralph mode is active."""
+    return Path(STATE_FILE).exists()
+
+
+def load_state():
+    """Load Ralph state."""
+    with open(STATE_FILE) as f:
+        return json.load(f)
+
+
+def save_state(state):
+    """Save Ralph state."""
+    with open(STATE_FILE, "w") as f:
+        json.dump(state, f, indent=2)
+
+
+def trigger_checkpoint(level, percent, state):
+    """Trigger a CCS checkpoint within Ralph iteration."""
+    timestamp = datetime.now(timezone.utc).isoformat()
+    handoff_file = f".claude/handoffs/{datetime.now().strftime('%Y-%m-%d-%H%M')}.md"
+
+    state["context_checkpoint"] = {
+        "active": True,
+        "level": level,
+        "percent_at_checkpoint": percent,
+        "timestamp": timestamp,
+        "handoff_file": handoff_file,
+        "resume_summary": "",  # Filled by handoff generation
+        "files_in_progress": [],
+        "current_todo_item": state.get("todo_items", [""])[0] if state.get("todo_items") else "",
+        "progress_on_item": "unknown"
+    }
+
+    # Add to context history
+    if "context_history" not in state:
+        state["context_history"] = []
+
+    state["context_history"].append({
+        "iteration": state.get("iteration", 0),
+        "peak_percent": percent,
+        "checkpoint": True
+    })
+
+    save_state(state)
+
+    # Signal that checkpoint should be generated
+    # This triggers the /handoff command with CCS metadata
+    print(f"CCS_CHECKPOINT_TRIGGERED|{level}|{percent}|{handoff_file}")
+
+
+def main():
+    if not is_ralph_active():
+        return  # Not in Ralph mode
+
+    percent = get_context_percent()
+    state = load_state()
+
+    # Track peak context for history
+    current_peak = state.get("context_history", [{}])[-1].get("peak_percent", 0)
+    if percent > current_peak:
+        # Update peak but don't trigger checkpoint yet
+        pass
+
+    # Check thresholds
+    if percent >= THRESHOLD_L3:
+        trigger_checkpoint("L3", percent, state)
+        sys.exit(1)  # Signal emergency stop
+
+    elif percent >= THRESHOLD_L2:
+        trigger_checkpoint("L2", percent, state)
+        # Don't exit yet - let current edit complete
+        # The checkpoint signal will be handled by Claude
+
+    elif percent >= THRESHOLD_L1:
+        # Just warn, don't checkpoint
+        print(f"CCS_WARNING|L1|{percent}|Context at {percent}%")
+
+
+if __name__ == "__main__":
+    main()
+```
+
+### ralph_stop.sh Enhancement
+
+Add CCS checkpoint handling:
+
+```bash
+# =============================================================================
+# NEW: Check for CCS Checkpoint
+# =============================================================================
+
+check_ccs_checkpoint() {
+    # Check if a context checkpoint was triggered this iteration
+    local checkpoint_active
+    checkpoint_active=$(jq -r '.context_checkpoint.active // false' "$STATE_FILE" 2>/dev/null)
+
+    if [[ "$checkpoint_active" == "true" ]]; then
+        return 0  # Checkpoint was triggered
+    fi
+
+    return 1
+}
+
+handle_ccs_checkpoint() {
+    log_event "INFO" "CCS checkpoint detected - preparing resume context"
+
+    # Read checkpoint data
+    local level percent handoff_file
+    level=$(jq -r '.context_checkpoint.level' "$STATE_FILE")
+    percent=$(jq -r '.context_checkpoint.percent_at_checkpoint' "$STATE_FILE")
+    handoff_file=$(jq -r '.context_checkpoint.handoff_file' "$STATE_FILE")
+
+    log_event "INFO" "Checkpoint: $level at $percent%, handoff: $handoff_file"
+
+    # Regenerate PROMPT.md with resume context
+    # This calls ralph_state.py to rebuild PROMPT.md with checkpoint data
+    python3 global/lib/ralph_state.py regenerate-prompt --checkpoint
+
+    # Reset checkpoint.active for next iteration
+    jq '.context_checkpoint.active = false' "$STATE_FILE" > tmp.$$ && mv tmp.$$ "$STATE_FILE"
+
+    # Continue loop
+    return 1  # Exit code 1 = restart loop
+}
+
+# In main() function, add before other checks:
+main() {
+    # ... existing init code ...
+
+    # NEW: Check for CCS checkpoint
+    if check_ccs_checkpoint; then
+        handle_ccs_checkpoint
+        exit 1  # Restart loop with checkpoint context
+    fi
+
+    # ... rest of existing checks ...
+}
+```
+
+---
+
+## Checkpoint Flow Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│              CCS CHECKPOINT WITHIN RALPH ITERATION                   │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  Context at 85% (L2 threshold)                                      │
+│  │                                                                  │
+│  ▼                                                                  │
+│  ralph_context_monitor.py fires                                     │
+│  │                                                                  │
+│  ├── 1. Set context_checkpoint.active = true in state               │
+│  ├── 2. Record level, percent, timestamp                            │
+│  ├── 3. Output: CCS_CHECKPOINT_TRIGGERED|L2|85|handoff.md           │
+│  │                                                                  │
+│  ▼                                                                  │
+│  Claude Code receives signal                                        │
+│  │                                                                  │
+│  ├── 4. Complete current edit (max 60s)                             │
+│  ├── 5. Run /handoff with CCS metadata                              │
+│  │      └── Generates .claude/handoffs/2026-01-07-1145.md           │
+│  ├── 6. Commit WIP: "ralph: checkpoint iter 5 (L2 at 85%)"          │
+│  ├── 7. Update Linear issue with checkpoint note                    │
+│  └── 8. End session                                                 │
+│                                                                     │
+│  ▼                                                                  │
+│  ralph_stop.sh fires                                                │
+│  │                                                                  │
+│  ├── 9. Detect context_checkpoint.active = true                     │
+│  ├── 10. Call handle_ccs_checkpoint()                               │
+│  │       ├── Regenerate PROMPT.md with resume context               │
+│  │       └── Reset checkpoint.active = false                        │
+│  └── 11. exit 1 (restart loop)                                      │
+│                                                                     │
+│  ▼                                                                  │
+│  ralph-loop.sh continues                                            │
+│  │                                                                  │
+│  └── 12. Start iteration 6 with fresh context + resume instructions │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Context Efficiency Strategies for Ralph
+
+### Strategy 1: Aggressive Subagent Delegation
+
+Ralph already recommends subagents for research. Enhance this:
+
+| Task Type | Subagent | Context Savings |
+|-----------|----------|-----------------|
+| File exploration | Explore agent | ~20% |
+| Code review | Code-reviewer agent | ~15% |
+| Test analysis | spec-tester agent | ~15% |
+| Documentation | docs-architect agent | ~10% |
+
+### Strategy 2: Incremental Context Loading
+
+Instead of loading all context at start, load incrementally:
+
+```markdown
+## PROMPT.md - Iteration 5
+
+### Minimal Context (Always Load)
+- fix_plan.md (TODO list)
+- progress.txt (last 2 iterations only)
+- checkpoint resume data (if any)
+
+### Load On Demand
+- Full file contents: Only when editing
+- Previous handoffs: Only if resuming from checkpoint
+- Spec files: Only if referenced in current TODO item
+```
+
+### Strategy 3: Context Budget Per TODO Item
+
+Estimate context per item:
+
+| Item Complexity | Estimated Context | Max Per Iteration |
+|-----------------|-------------------|-------------------|
+| Trivial (<50 LOC) | 5-10% | 8-10 items |
+| Simple (50-150 LOC) | 10-20% | 4-5 items |
+| Medium (150-300 LOC) | 20-40% | 2-3 items |
+| Complex (>300 LOC) | 40-60% | 1 item |
+| Very Complex | 60%+ | MUST shard |
+
+If estimated context for item exceeds remaining context, checkpoint BEFORE starting item.
+
+---
+
+## New Commands
+
+### /ralph checkpoint
+
+Manual checkpoint within Ralph:
+
+```
+/ralph checkpoint
+
+## Ralph Checkpoint Created
+
+**Iteration**: 5
+**Context**: 72%
+**Handoff**: .claude/handoffs/2026-01-07-1200.md
+
+Next iteration will resume from this point.
+Continue with current iteration? (will restart on next natural stop)
+```
+
+### /ralph context
+
+Show context status within Ralph:
+
+```
+/ralph context
+
+## Ralph Context Status
+
+| Metric | Value |
+|--------|-------|
+| Current Context | 67% |
+| Level | L0 (Normal) |
+| Estimated Remaining | ~66k tokens |
+| Current Item | "Implement OAuth callback" |
+| Item Complexity | Medium (est. 25% context) |
+
+**Recommendation**: Safe to continue.
+Checkpoint will trigger automatically at 85%.
+```
+
+---
+
+## Success Criteria
+
+| Metric | Target |
+|--------|--------|
+| Context loss per Ralph run | <5% (vs 100% without CCS) |
+| Successful checkpoint resumes | >95% |
+| Iterations per context window | Increase by 30% |
+| Task continuity across checkpoints | 100% (no lost work) |
+| Overnight run success rate | >90% (vs ~60% without CCS) |
+
+---
+
+## Implementation Priority
+
+### Phase 1: Core Integration (P0)
+1. Add `context_checkpoint` fields to ralph-state.json
+2. Create `ralph_context_monitor.py` PostToolUse hook
+3. Enhance `ralph_stop.sh` with CCS checkpoint handling
+4. Modify PROMPT.md generation to include resume context
+
+### Phase 2: Automatic Checkpointing (P0)
+1. Implement L2 automatic checkpoint trigger
+2. Generate CCS-enhanced handoff documents
+3. Test checkpoint → resume cycle
+
+### Phase 3: Context Efficiency (P1)
+1. Implement subagent delegation recommendations
+2. Add context budget estimation per TODO item
+3. Add `/ralph context` status command
+
+### Phase 4: Intelligence (P2)
+1. Learn context patterns from history
+2. Predict optimal checkpoint timing
+3. Recommend task reordering for efficiency
+
+---
+
+## Open Questions
+
+1. **Context percentage API**: How does PostToolUse hook get context percentage from Claude Code?
+   - Option A: Environment variable set by Claude Code
+   - Option B: Parse from statusline output
+   - Option C: Query Claude Code API
+
+2. **Checkpoint granularity**: Should we checkpoint mid-file or only at file boundaries?
+   - Recommendation: File boundaries when possible, mid-file only at L3
+
+3. **Linear integration**: Should Ralph iterations create separate Linear comments or append to existing?
+   - Recommendation: Append with iteration number prefix
+
+4. **Handoff document size**: How much detail in checkpoint handoff?
+   - Recommendation: Structured summary + references, not full context dump
+
+---
+
+*This design document prepared 2026-01-07 for CCS-Ralph integration planning.*