the-grid-cc 1.7.14 → 1.7.16

package/docs/PERSISTENCE_IMPLEMENTATION.md

@@ -0,0 +1,361 @@
+# Grid Persistence - Implementation Guide
+
+**Version:** 1.0
+**Date:** 2026-01-23
+
+This guide explains how to implement and use Grid's persistence system.
+
+## Quick Start
+
+### Initialize a Project
+
+```bash
+/grid:init
+```
+
+This creates the complete `.grid/` directory structure with all state files.
+
+### Resume an Interrupted Mission
+
+```bash
+/grid:resume
+```
+
+This reconstructs context from `.grid/` files and continues from the last checkpoint.
+
+## Implementation Status
+
+### Phase 1: Core Persistence (REQUIRED)
+
+- [x] Template files created
+- [x] `/grid:resume` added to help
+- [ ] STATE.md write on every wave complete
+- [ ] CHECKPOINT.md write on checkpoint/interrupt
+- [ ] WARMTH.md aggregation after block complete
+- [ ] `/grid:resume` command full implementation
+- [ ] Basic context reconstruction
+
+### Phase 2: Enhanced Recovery
+
+- [ ] Session death detection via scratchpad staleness
+- [ ] Git-based state reconstruction
+- [ ] Corrupted state recovery
+- [ ] Rollback support
+
+### Phase 3: Advanced Features
+
+- [ ] Multi-cluster support
+- [ ] State diff visualization
+- [ ] Time-travel debugging
+- [ ] Cross-session analytics
+
+## File Templates Location
+
+All templates are in: `/Users/jacweath/grid/templates/grid-state/`
+
+- `STATE.md` - Central state tracking
+- `WARMTH.md` - Institutional knowledge
+- `SCRATCHPAD.md` - Live discoveries
+- `DECISIONS.md` - User decisions
+- `BLOCKERS.md` - Blocker tracking
+- `CHECKPOINT.md` - Interrupted thread state
+- `config.json` - Grid configuration
+- `BLOCK-SUMMARY.md` - Completed block record
+
+## Integration Points
+
+### Master Control (mc.md)
+
+Master Control must:
+
+1. **On wave complete:**
+   - Update STATE.md with new position
+   - Update progress_percent
+   - Update updated_at timestamp
+
+2. **On block complete:**
+   - Aggregate WARMTH.md from executor's lessons_learned
+   - Verify SUMMARY.md was written
+   - Update STATE.md
+
+3. **On checkpoint:**
+   - Write CHECKPOINT.md with current thread state
+   - Set STATE.md status to "checkpoint"
+   - Wait for user response
+
+4. **On session approaching exhaustion:**
+   - Write CHECKPOINT.md with type: session_death
+   - Set STATE.md status to "interrupted"
+   - Include partial_work details
+
+### Grid Executor (grid-executor.md)
+
+Executors must:
+
+1. **On thread complete:**
+   - Commit work with clear message
+   - Record commit hash
+
+2. **On block complete:**
+   - Write SUMMARY.md to `.grid/phases/{phase}/`
+   - Include all commits with hashes
+   - Include lessons_learned for warmth aggregation
+   - List all artifacts_created
+
+3. **On scratchpad entry:**
+   - Write discovery to SCRATCHPAD.md
+   - Include timestamp and program-id
+   - Follow standard format
+
+4. **On blocker encountered:**
+   - Write to BLOCKERS.md
+   - Include type, description, position
+   - Mark as ACTIVE
+
+### Resume Command (resume.md)
+
+The `/grid:resume` command must:
+
+1. **Detect state:**
+   - Check if STATE.md exists
+   - Parse status and position
+   - Determine resume strategy
+
+2. **Validate state:**
+   - Verify commits exist in git
+   - Verify claimed files exist
+   - Check for conflicts
+
+3. **Reconstruct context:**
+   - Load STATE.md
+   - Load WARMTH.md
+   - Load DECISIONS.md
+   - Load CHECKPOINT.md if exists
+   - Collect all SUMMARY.md files
+   - Build execution context
+
+4. **Spawn continuation:**
+   - Inject warmth into executor prompt
+   - Provide completed_threads table
+   - Provide pending plan
+   - Set resume_point
+
+## State Update Protocol
+
+### Atomic Updates
+
+Always use atomic writes to prevent corruption:
+
+```python
+# 1. Read current state
+current = parse_yaml(read(".grid/STATE.md"))
+
+# 2. Apply updates
+merged = deep_merge(current, updates)
+merged["updated_at"] = datetime.now().isoformat()
+
+# 3. Write to temp file
+write(".grid/STATE.md.tmp", to_yaml(merged))
+
+# 4. Atomic rename
+rename(".grid/STATE.md.tmp", ".grid/STATE.md")
+```
+
+### Update Triggers
+
+| Event | File | Field Updated |
+|-------|------|---------------|
+| Wave starts | STATE.md | status: active, wave: N |
+| Wave completes | STATE.md | wave: N+1, progress_percent |
+| Block completes | STATE.md | block: N+1 |
+| Checkpoint hit | STATE.md, CHECKPOINT.md | status: checkpoint |
+| Session ending | STATE.md, CHECKPOINT.md | status: interrupted |
+| Mission complete | STATE.md | status: completed |
+| Blocker found | BLOCKERS.md | New entry |
+| User decision | DECISIONS.md | New entry |
+| Discovery made | SCRATCHPAD.md | New entry |
+
+## Warmth Aggregation
+
+After each block completes, aggregate warmth:
+
+```python
+def aggregate_warmth(block_summary_path):
+    # 1. Parse block summary
+    summary = parse_yaml(read(block_summary_path))
+    block_warmth = summary.get("lessons_learned", {})
+
+    # 2. Load existing warmth
+    if file_exists(".grid/WARMTH.md"):
+        existing = parse_yaml(read(".grid/WARMTH.md"))
+    else:
+        existing = {
+            "codebase_patterns": [],
+            "gotchas": [],
+            "user_preferences": [],
+            "decisions_made": [],
+            "almost_did": [],
+        }
+
+    # 3. Merge (deduplicate)
+    for category in ["codebase_patterns", "gotchas", "user_preferences", "almost_did"]:
+        new_items = block_warmth.get(category, [])
+        for item in new_items:
+            if item not in existing[category]:
+                existing[category].append(item)
+
+    # 4. Write aggregated warmth
+    write(".grid/WARMTH.md", to_yaml(existing))
+```
+
+## Context Reconstruction
+
+When resuming, rebuild complete context:
+
+```python
+def reconstruct_context():
+    context = {
+        "cluster": None,
+        "position": None,
+        "completed_blocks": [],
+        "completed_threads": [],
+        "pending_plans": [],
+        "warmth": None,
+        "decisions": [],
+        "blockers": [],
+        "checkpoint": None,
+    }
+
+    # 1. Load central state
+    state = parse_yaml(read(".grid/STATE.md"))
+    context["cluster"] = state["cluster"]
+    context["position"] = state["position"]
+
+    # 2. Collect completed work
+    for summary_path in glob(".grid/phases/*/SUMMARY.md"):
+        summary = parse_yaml(read(summary_path))
+        context["completed_blocks"].append(summary)
+
+    # 3. Load warmth
+    if file_exists(".grid/WARMTH.md"):
+        context["warmth"] = read(".grid/WARMTH.md")
+
+    # 4. Load decisions
+    if file_exists(".grid/DECISIONS.md"):
+        context["decisions"] = parse_decisions(".grid/DECISIONS.md")
+
+    # 5. Load checkpoint if exists
+    if file_exists(".grid/CHECKPOINT.md"):
+        context["checkpoint"] = parse_yaml(read(".grid/CHECKPOINT.md"))
+
+    # 6. Identify pending plans
+    for plan_path in glob(".grid/plans/*-block-*.md"):
+        block_num = extract_block_number(plan_path)
+        if block_num not in [b["block"] for b in context["completed_blocks"]]:
+            context["pending_plans"].append({
+                "path": plan_path,
+                "block": block_num,
+                "content": read(plan_path),
+            })
+
+    return context
+```
+
+## Checkpoint Types
+
+### Human Verify Checkpoint
+
+```yaml
+type: human_verify
+checkpoint_details:
+  verification_instructions: |
+    1. Run: npm run dev
+    2. Visit: http://localhost:4321
+    3. Click dark mode toggle
+    4. Verify theme persists on refresh
+awaiting: "User to respond 'approved' or describe issues"
+```
+
+### Decision Checkpoint
+
+```yaml
+type: decision
+checkpoint_details:
+  question: "Deploy to Vercel or Netlify?"
+  options:
+    - id: vercel
+      description: "Native Astro support, edge functions"
+    - id: netlify
+      description: "Simpler config, build plugins"
+awaiting: "User to choose option"
+```
+
+### Human Action Checkpoint
+
+```yaml
+type: human_action
+checkpoint_details:
+  required_action: "Run: vercel login"
+  reason: "Vercel CLI authentication needed"
+awaiting: "User to complete action and confirm"
+```
+
+### Session Death Checkpoint
+
+```yaml
+type: session_death
+checkpoint_details:
+  last_action: "Writing localStorage persistence logic"
+  partial_work:
+    files_created: ["src/components/DarkModeToggle.astro"]
+    files_modified: ["src/layouts/BaseLayout.astro"]
+    staged_changes: true
+awaiting: "Automatic resume on next session"
+```
+
+## Testing Persistence
+
+### Test 1: Clean Checkpoint Resume
+
+1. Start mission: `/grid`
+2. Wait for checkpoint
+3. Approve checkpoint
+4. Close terminal (simulate session death)
+5. New session: `/grid:resume`
+6. **Expected:** Continues from approved checkpoint
+
+### Test 2: Session Death Recovery
+
+1. Start mission: `/grid`
+2. During execution, kill terminal (SIGKILL)
+3. New session: `/grid:resume`
+4. **Expected:** Detects stale state, reconstructs from scratchpad + git
+
+### Test 3: Failure Recovery
+
+1. Start mission that will fail (e.g., missing API key)
+2. Executor returns failure
+3. New session: `/grid:resume`
+4. **Expected:** Presents failure report with recovery options
+
+## Next Steps
+
+1. **Implement STATE.md updates in mc.md**
+   - Add wave complete handler
+   - Add block complete handler
+   - Add checkpoint handler
+
+2. **Implement SUMMARY.md writes in grid-executor.md**
+   - Add block complete handler
+   - Include lessons_learned section
+
+3. **Complete /grid:resume implementation**
+   - Add state validation
+   - Add context reconstruction
+   - Add continuation spawning
+
+4. **Test end-to-end persistence**
+   - Run full mission with interruption
+   - Verify resume works correctly
+
+End of Line.

package/docs/PERSISTENCE_QUICKSTART.md

@@ -0,0 +1,283 @@
+# Grid Persistence - Quick Start Guide
+
+**Version:** 1.0
+**Status:** Ready for Implementation
+
+This guide provides a quick overview of Grid's persistence system.
+
+## What is Grid Persistence?
+
+Grid Persistence allows missions to survive session death. When your session ends unexpectedly (context exhaustion, timeout, disconnect), all state is preserved in `.grid/` files. A fresh Master Control can resume exactly where you left off.
+
+## Core Concepts
+
+### State Files
+
+| File | Purpose | When Written |
+|------|---------|--------------|
+| `STATE.md` | Central position tracking | Every wave |
+| `WARMTH.md` | Accumulated knowledge | After each block |
+| `CHECKPOINT.md` | Interrupted thread state | On checkpoint/interrupt |
+| `SCRATCHPAD.md` | Live discoveries | During execution |
+| `DECISIONS.md` | User decisions | When user decides |
+| `BLOCKERS.md` | Issues blocking progress | When blocker encountered |
+
+### Directory Structure
+
+```
+.grid/
+├── STATE.md                    # Read this first on resume
+├── WARMTH.md                   # Knowledge accumulation
+├── SCRATCHPAD.md               # Live discoveries
+├── DECISIONS.md                # User decisions
+├── BLOCKERS.md                 # Active blockers
+├── config.json                 # Grid configuration
+│
+├── plans/                      # Execution plans
+├── phases/                     # Completed work
+├── discs/                      # Program Identity Discs
+├── debug/                      # Debug sessions
+└── refinement/                 # Refinement outputs
+```
+
+## Commands
+
+### Initialize State
+
+```bash
+/grid:init
+```
+
+Creates the `.grid/` directory with all state files.
+
+### Resume Mission
+
+```bash
+/grid:resume              # Auto-detect and resume
+/grid:resume --validate   # Validate state only
+/grid:resume --from block-02  # Resume from specific block
+```
+
+Reconstructs context from `.grid/` files and continues execution.
+
+### Check Status
+
+```bash
+/grid:status
+```
+
+Shows current mission state and progress.
+
+## How It Works
+
+### 1. During Execution
+
+- **Master Control** updates STATE.md after each wave
+- **Executors** write to SCRATCHPAD.md during work
+- **Executors** write SUMMARY.md after each block
+- **Master Control** aggregates WARMTH.md after each block
+
+### 2. On Checkpoint
+
+When execution hits a checkpoint (human verification, decision point, etc.):
+
+1. Executor writes CHECKPOINT.md with current thread state
+2. MC updates STATE.md status to "checkpoint"
+3. System waits for user response
+
+### 3. On Session Death
+
+If session dies unexpectedly:
+
+1. Last STATE.md update shows position
+2. SCRATCHPAD.md shows last activity
+3. Git history shows last commits
+4. Resume reconstructs state from these
+
+### 4. On Resume
+
+```
+/grid:resume
+
+1. Read STATE.md
+2. Validate state consistency
+3. Load WARMTH.md, DECISIONS.md, CHECKPOINT.md
+4. Collect all SUMMARY.md files
+5. Build execution context
+6. Spawn continuation with full context
+```
+
+## Resume Scenarios
+
+### Scenario 1: Clean Checkpoint
+
+**State:** User approved checkpoint, session ended normally
+
+```
+STATUS: checkpoint
+CHECKPOINT.md: exists with user_response
+```
+
+**Action:** Continue from next thread after checkpoint
+
+### Scenario 2: Session Death
+
+**State:** Session died mid-execution
+
+```
+STATUS: active (stale timestamp)
+CHECKPOINT.md: may not exist
+SCRATCHPAD.md: recent entries
+```
+
+**Action:** Reconstruct from scratchpad + git, resume from last known point
+
+### Scenario 3: Failure
+
+**State:** Executor failed with error
+
+```
+STATUS: failed
+CHECKPOINT.md: type: failure with details
+```
+
+**Action:** Present failure report, offer rollback/retry/manual options
+
+## Warmth Accumulation
+
+Warmth is institutional knowledge that survives across sessions:
+
+```yaml
+# WARMTH.md
+codebase_patterns:
+  - "This project uses Astro content collections"
+  - "Dark mode uses class strategy with localStorage"
+
+gotchas:
+  - "Astro config must use .mjs extension for ESM"
+  - "Shiki syntax highlighting is build-time only"
+
+user_preferences:
+  - "User prefers minimal dependencies"
+  - "User wants dark mode as default"
+
+decisions_made:
+  - "Chose serverless adapter for future flexibility"
+
+almost_did:
+  - "Considered MDX but stuck with plain MD"
+```
+
+When resuming, this warmth is injected into the continuation executor so it:
+- Doesn't repeat mistakes
+- Applies learned patterns
+- Respects user preferences
+
+## State Validation
+
+Before resuming, Grid validates:
+
+1. **Commits exist** - All claimed commit hashes are in git
+2. **Files exist** - All claimed artifacts exist on disk
+3. **Plans available** - Plans exist for pending blocks
+4. **Single checkpoint** - No conflicting checkpoint files
+5. **State parseable** - All YAML frontmatter is valid
+
+If validation fails, Grid enters recovery mode and attempts to reconstruct state from git + artifacts.
+
+## Templates
+
+All templates are in: `templates/grid-state/`
+
+- `STATE.md` - Central state template
+- `WARMTH.md` - Warmth template
+- `SCRATCHPAD.md` - Scratchpad template
+- `DECISIONS.md` - Decisions template
+- `BLOCKERS.md` - Blockers template
+- `CHECKPOINT.md` - Checkpoint template
+- `config.json` - Configuration template
+- `BLOCK-SUMMARY.md` - Block summary template
+
+## Example Flow
+
+```
+1. User: /grid
+   MC: Initializes .grid/, starts mission
+
+2. MC spawns Executor for block 01
+   Executor: Works on threads, commits, writes SCRATCHPAD.md
+
+3. Executor completes block 01
+   Executor: Writes SUMMARY.md with commits and lessons_learned
+   MC: Aggregates WARMTH.md from lessons_learned
+   MC: Updates STATE.md (block: 2, wave: 1, progress: 16%)
+
+4. Executor hits checkpoint for human verification
+   Executor: Writes CHECKPOINT.md (type: human_verify)
+   MC: Updates STATE.md (status: checkpoint)
+   MC: Waits for user
+
+5. Session dies (context exhaustion)
+   STATE.md: status: checkpoint (last write)
+   CHECKPOINT.md: awaiting user response
+
+6. New session
+   User: /grid:resume
+   MC: Reads STATE.md (status: checkpoint)
+   MC: Reads CHECKPOINT.md (type: human_verify)
+   MC: Loads WARMTH.md (accumulated knowledge)
+   MC: Presents checkpoint to user
+
+7. User approves checkpoint
+   MC: Updates CHECKPOINT.md (user_response: approved)
+   MC: Spawns continuation executor with:
+      - Warmth injected
+      - Completed threads table
+      - Resume point (next thread after checkpoint)
+
+8. Execution continues
+   Executor: Picks up from thread after checkpoint
+   Executor: Has full context from warmth
+   Executor: Doesn't repeat mistakes from prior attempts
+```
+
+## Implementation Status
+
+### Completed
+
+- [x] Template files created
+- [x] `/grid:resume` added to help
+- [x] resume.md spec complete
+- [x] init.md enhanced with templates
+- [x] PERSISTENCE.md design document
+- [x] Documentation complete
+
+### Next Steps
+
+1. **Implement state updates in mc.md**
+   - Wave complete handler
+   - Block complete handler
+   - Checkpoint handler
+
+2. **Implement summary writes in grid-executor.md**
+   - Block complete handler
+   - lessons_learned section
+
+3. **Complete /grid:resume implementation**
+   - State validation logic
+   - Context reconstruction logic
+   - Continuation spawning logic
+
+4. **Test end-to-end**
+   - Full mission with interruption
+   - Verify resume works correctly
+
+## Related Documentation
+
+- `/docs/PERSISTENCE.md` - Full technical design
+- `/docs/PERSISTENCE_IMPLEMENTATION.md` - Implementation guide
+- `/templates/grid-state/README.md` - Template documentation
+- `/commands/grid/resume.md` - Resume command spec
+- `/commands/grid/init.md` - Init command spec
+
+End of Line.