the-grid-cc 1.7.14 → 1.7.16

package/docs/PERSISTENCE_FLOW.md

@@ -0,0 +1,483 @@
+# Grid Persistence - Checkpoint & Resume Flow
+
+**Version:** 1.0
+**Visual Guide**
+
+This document visualizes the checkpoint and resume flow in The Grid.
+
+## State Lifecycle
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                         GRID STATE LIFECYCLE                        │
+└─────────────────────────────────────────────────────────────────────┘
+
+  INITIALIZED ──▶ ACTIVE ──▶ CHECKPOINT ──▶ INTERRUPTED ──▶ RESUMED
+      │             │           │              │               │
+      │             │           │              │               ▼
+      │             │           │              │          ACTIVE
+      │             │           │              │               │
+      │             │           │              │               ▼
+      │             │           │              └──────▶   COMPLETED
+      │             │           │
+      │             │           └────────────────────────▶ FAILED
+      │             │
+      │             └──────────────────────────────────▶ COMPLETED
+      │
+      └──────────────────────────────────────────────▶ (never started)
+```
+
+## File Write Timeline
+
+```
+Time ──────────────────────────────────────────────────────────▶
+
+/grid:init
+   │
+   ├─ STATE.md (status: initialized)
+   ├─ WARMTH.md (empty)
+   ├─ SCRATCHPAD.md (empty)
+   ├─ DECISIONS.md (empty)
+   ├─ BLOCKERS.md (empty)
+   └─ config.json
+
+/grid "build blog"
+   │
+   ├─ STATE.md (status: active, cluster: blog)
+   └─ plans/ (block plans written)
+
+Block 01 execution starts
+   │
+   ├─ SCRATCHPAD.md (executor writes discoveries)
+   │   ├─ executor-01 | 14:00:00 | PATTERN
+   │   │   Finding: "Astro uses .astro extension"
+   │   │
+   │   ├─ executor-01 | 14:05:00 | PROGRESS
+   │   │   Finding: "Thread 1 complete (commit abc123)"
+   │   │
+   │   └─ executor-01 | 14:10:00 | GOTCHA
+   │       Finding: "Config must be .mjs for ESM"
+
+Block 01 completes
+   │
+   ├─ phases/01-foundation/01-SUMMARY.md (complete block record)
+   ├─ WARMTH.md (aggregated from lessons_learned)
+   └─ STATE.md (block: 2, progress: 16%)
+
+Block 02 execution starts
+   │
+   └─ SCRATCHPAD.md (new entries)
+
+Thread 02.3 hits checkpoint
+   │
+   ├─ CHECKPOINT.md (type: human_verify)
+   └─ STATE.md (status: checkpoint)
+
+Session dies (context exhaustion)
+   │
+   └─ (last STATE.md write: status: checkpoint)
+
+/grid:resume (new session)
+   │
+   ├─ Read STATE.md (status: checkpoint)
+   ├─ Read CHECKPOINT.md (awaiting user)
+   ├─ Read WARMTH.md (accumulated knowledge)
+   └─ Present checkpoint to user
+
+User responds "approved"
+   │
+   ├─ CHECKPOINT.md (user_response: approved)
+   ├─ STATE.md (status: active)
+   └─ Spawn continuation executor
+
+Execution continues
+   │
+   └─ (normal flow resumes)
+```
+
+## Checkpoint Types Flow
+
+### Human Verify Checkpoint
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                     HUMAN VERIFY CHECKPOINT                         │
+└─────────────────────────────────────────────────────────────────────┘
+
+Executor completes thread requiring verification
+   │
+   ├─ Writes CHECKPOINT.md
+   │   ├─ type: human_verify
+   │   ├─ verification_instructions
+   │   ├─ expected_behavior
+   │   └─ completed_threads (with commits)
+   │
+   ├─ Updates STATE.md (status: checkpoint)
+   │
+   └─ Returns to MC
+
+MC presents checkpoint to user
+   │
+   └─ "Please test dark mode and respond 'approved' or describe issues"
+
+User tests and responds
+   │
+   ├─ "approved" ──▶ Continue from next thread
+   │
+   └─ "toggle doesn't persist" ──▶ Fix issue, retry thread
+
+Session may die here ──▶ Resume reconstructs this state
+```
+
+### Decision Checkpoint
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      DECISION CHECKPOINT                            │
+└─────────────────────────────────────────────────────────────────────┘
+
+Executor reaches architectural fork
+   │
+   ├─ Writes CHECKPOINT.md
+   │   ├─ type: decision
+   │   ├─ question: "Deploy to Vercel or Netlify?"
+   │   ├─ options: [vercel, netlify]
+   │   └─ affects: "Block 01, Block 06"
+   │
+   ├─ Updates STATE.md (status: checkpoint)
+   │
+   └─ Returns to MC
+
+MC presents decision to user
+   │
+   └─ Options: 1) Vercel, 2) Netlify
+
+User chooses
+   │
+   ├─ CHECKPOINT.md (user_response: "vercel")
+   ├─ DECISIONS.md (new entry)
+   └─ Continue with chosen path
+```
+
+### Session Death Checkpoint
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                    SESSION DEATH CHECKPOINT                         │
+└─────────────────────────────────────────────────────────────────────┘
+
+Executor detects context approaching exhaustion
+   │
+   ├─ Writes CHECKPOINT.md
+   │   ├─ type: session_death
+   │   ├─ last_action: "Writing localStorage logic"
+   │   ├─ partial_work: {files, changes}
+   │   └─ completed_threads
+   │
+   ├─ Updates STATE.md (status: interrupted)
+   │
+   └─ Session ends
+
+/grid:resume (new session)
+   │
+   ├─ Detects: status: interrupted
+   ├─ Reads CHECKPOINT.md (type: session_death)
+   ├─ Validates partial_work
+   │
+   ├─ If good ──▶ Commit and continue
+   └─ If broken ──▶ Rollback and retry thread
+```
+
+### Failure Checkpoint
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      FAILURE CHECKPOINT                             │
+└─────────────────────────────────────────────────────────────────────┘
+
+Executor encounters unrecoverable error
+   │
+   ├─ Writes CHECKPOINT.md
+   │   ├─ type: failure
+   │   ├─ error: "Vercel API key not found"
+   │   ├─ partial_work: {what was done}
+   │   └─ completed_threads
+   │
+   ├─ Updates STATE.md (status: failed)
+   │
+   └─ Returns to MC
+
+MC presents failure to user
+   │
+   └─ Options:
+       1) Retry (spawn fresh executor)
+       2) Rollback (revert partial work)
+       3) Manual (user fixes, then continue)
+       4) Skip (skip this block)
+
+User chooses
+   │
+   ├─ Retry ──▶ Spawn executor with failure context
+   ├─ Rollback ──▶ git reset --hard, restart block
+   ├─ Manual ──▶ Wait for user to fix, then continue
+   └─ Skip ──▶ Mark block as skipped, continue to next
+```
+
+## Resume Context Reconstruction
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                    CONTEXT RECONSTRUCTION                           │
+└─────────────────────────────────────────────────────────────────────┘
+
+/grid:resume
+   │
+   ├─ Step 1: State Detection
+   │   ├─ Read STATE.md
+   │   ├─ Check status: checkpoint | interrupted | failed
+   │   └─ Determine resume strategy
+   │
+   ├─ Step 2: State Validation
+   │   ├─ Verify commits exist in git ✓
+   │   ├─ Verify claimed files exist ✓
+   │   ├─ Check for conflicts ✓
+   │   └─ If invalid ──▶ Enter recovery mode
+   │
+   ├─ Step 3: Context Reconstruction
+   │   ├─ Load WARMTH.md
+   │   │   └─ Accumulated knowledge from all prior Programs
+   │   │
+   │   ├─ Load DECISIONS.md
+   │   │   └─ User decisions that affect future work
+   │   │
+   │   ├─ Load CHECKPOINT.md (if exists)
+   │   │   └─ Exact thread state when interrupted
+   │   │
+   │   ├─ Collect SUMMARY.md files
+   │   │   └─ All completed blocks with commits
+   │   │
+   │   └─ Load pending PLAN.md
+   │       └─ Remaining work to be done
+   │
+   ├─ Step 4: Build Executor Prompt
+   │   │
+   │   ├─ <warmth>
+   │   │   {accumulated knowledge}
+   │   │   </warmth>
+   │   │
+   │   ├─ <completed_threads>
+   │   │   - Thread 02.1: BaseLayout (commit abc123)
+   │   │   - Thread 02.2: Header (commit def456)
+   │   │   </completed_threads>
+   │   │
+   │   ├─ <resume_point>
+   │   │   Continue from: Thread 02.4
+   │   │   User feedback: {if any}
+   │   │   </resume_point>
+   │   │
+   │   └─ <plan>
+   │       {remaining threads from PLAN.md}
+   │       </plan>
+   │
+   └─ Step 5: Spawn Continuation
+       ├─ Inject warmth ──▶ Executor has institutional knowledge
+       ├─ Provide completed_threads ──▶ Executor knows what's done
+       ├─ Set resume_point ──▶ Executor starts from correct thread
+       └─ Execute ──▶ Picks up exactly where prior session left off
+```
+
+## Warmth Aggregation Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      WARMTH AGGREGATION                             │
+└─────────────────────────────────────────────────────────────────────┘
+
+Block 01 completes
+   │
+   └─ 01-SUMMARY.md written with lessons_learned:
+       ├─ codebase_patterns: ["Astro uses content collections"]
+       ├─ gotchas: ["Config must be .mjs"]
+       └─ user_preferences: ["Minimal dependencies"]
+
+MC aggregates warmth
+   │
+   ├─ Read 01-SUMMARY.md lessons_learned
+   ├─ Read existing WARMTH.md (if exists)
+   ├─ Merge (deduplicate)
+   │   ├─ codebase_patterns: +1 new pattern
+   │   ├─ gotchas: +1 new gotcha
+   │   └─ user_preferences: +1 new preference
+   └─ Write updated WARMTH.md
+
+Block 02 starts
+   │
+   └─ Executor receives WARMTH.md in prompt
+       ├─ Knows about content collections
+       ├─ Avoids .js config mistake
+       └─ Respects minimal dependencies preference
+
+Block 02 completes
+   │
+   └─ 02-SUMMARY.md written with MORE lessons_learned:
+       ├─ codebase_patterns: ["Dark mode uses class strategy"]
+       ├─ gotchas: ["Must inline theme script to prevent FOUC"]
+       └─ decisions_made: ["Chose dark as default"]
+
+MC aggregates warmth again
+   │
+   ├─ Read 02-SUMMARY.md lessons_learned
+   ├─ Read existing WARMTH.md (has Block 01 warmth)
+   ├─ Merge (deduplicate)
+   │   ├─ codebase_patterns: +1 (now has 2)
+   │   ├─ gotchas: +1 (now has 2)
+   │   ├─ user_preferences: (still has 1)
+   │   └─ decisions_made: +1 (new category)
+   └─ Write updated WARMTH.md
+
+Block 03 starts
+   │
+   └─ Executor receives WARMTH.md with ALL prior knowledge
+       └─ Benefits from all discoveries in Blocks 01 and 02
+
+Session dies
+   │
+   └─ WARMTH.md persists on disk
+
+/grid:resume
+   │
+   └─ Continuation executor receives WARMTH.md
+       └─ Has full institutional knowledge
+       └─ Doesn't repeat mistakes
+       └─ Applies learned patterns
+```
+
+## State Validation Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      STATE VALIDATION                               │
+└─────────────────────────────────────────────────────────────────────┘
+
+/grid:resume
+   │
+   └─ Validation Phase
+
+Check 1: Commits exist
+   │
+   ├─ For each SUMMARY.md:
+   │   ├─ Extract commit hashes
+   │   └─ Run: git cat-file -t {hash}
+   │
+   ├─ All exist ──▶ [PASS]
+   └─ Missing ──▶ [FAIL] "Commit abc123 not found"
+
+Check 2: Files exist
+   │
+   ├─ For each SUMMARY.md:
+   │   ├─ Extract artifacts_created paths
+   │   └─ Check file_exists(path)
+   │
+   ├─ All exist ──▶ [PASS]
+   └─ Missing ──▶ [FAIL] "File src/layouts/BaseLayout.astro missing"
+
+Check 3: Plans available
+   │
+   ├─ For each pending block:
+   │   └─ Check: .grid/plans/*-block-{N}.md exists
+   │
+   ├─ All exist ──▶ [PASS]
+   └─ Missing ──▶ [FAIL] "Plan for block 03 missing"
+
+Check 4: Single checkpoint
+   │
+   ├─ Count CHECKPOINT*.md files
+   │
+   ├─ 0 or 1 ──▶ [PASS]
+   └─ Multiple ──▶ [FAIL] "Multiple checkpoint files found"
+
+Check 5: State parseable
+   │
+   ├─ Try: parse_yaml(STATE.md)
+   │
+   ├─ Success ──▶ [PASS]
+   └─ Error ──▶ [FAIL] "STATE.md corrupted"
+
+Results:
+   │
+   ├─ All PASS ──▶ State is valid, continue resume
+   │
+   └─ Any FAIL ──▶ Enter recovery mode
+       ├─ Attempt reconstruction from git + artifacts
+       ├─ Present reconstructed state to user
+       └─ User approves or aborts
+```
+
+## Recovery Mode Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                         RECOVERY MODE                               │
+└─────────────────────────────────────────────────────────────────────┘
+
+Validation failed or STATE.md corrupted
+   │
+   └─ Enter recovery mode
+
+Step 1: Scan for artifacts
+   │
+   ├─ Find all SUMMARY.md files
+   │   └─ Extract: blocks complete, commits
+   │
+   ├─ Scan git log for Grid commits
+   │   └─ Pattern: "feat(NN):" or "fix(NN):"
+   │
+   └─ Find all plan files
+       └─ Determine: total blocks planned
+
+Step 2: Reconstruct state
+   │
+   ├─ Completed blocks: [01, 02] (from SUMMARY.md)
+   ├─ Total commits: 15 (from git log)
+   ├─ Current block: 03 (inferred)
+   ├─ Progress: ~33% (estimated)
+   └─ Cluster: "blog" (inferred from plans)
+
+Step 3: Present to user
+   │
+   └─ Display:
+       """
+       STATE RECOVERY
+       ==============
+
+       Reconstructed state:
+       - Cluster: blog (inferred from plans)
+       - Completed: Blocks 01, 02
+       - Current: Block 03 (inferred)
+       - Progress: ~33%
+       - Commits: 15 found in git
+
+       Accept reconstructed state? (yes/no)
+       """
+
+Step 4: User decision
+   │
+   ├─ Yes ──▶ Write new STATE.md, continue resume
+   │
+   └─ No ──▶ Options:
+       1) Abort (clear state, start fresh)
+       2) Manual (user fixes STATE.md manually)
+       3) Inspect (show all found artifacts for manual review)
+```
+
+## Summary
+
+Grid's persistence system provides:
+
+1. **Checkpoint-based recovery** - Resume from exact stopping point
+2. **Warmth accumulation** - Knowledge transfers across sessions
+3. **State validation** - Ensures consistency before resume
+4. **Recovery mode** - Handles corrupted state gracefully
+5. **Multiple checkpoint types** - Handles various interruption scenarios
+
+End of Line.