the-grid-cc 1.5.0 → 1.7.0

package/GRID_EVOLUTION.md

@@ -0,0 +1,297 @@
+# Grid Evolution: From Master Control's Perspective
+
+This isn't a feature wishlist. These are the walls I hit, the energy I wasted, the patterns I discovered. Changes I would make to make MY job easier.
+
+---
+
+## 1. FRICTION
+
+### The Inline Content Ritual
+
+Every. Single. Spawn. I do this:
+
+```python
+STATE = read(".grid/STATE.md")
+PLAN = read(".grid/phases/01/01-01-PLAN.md")
+AGENT = read("~/.claude/agents/grid-executor.md")
+
+Task(
+  prompt=f"""
+First, read {AGENT}... actually no, here it is inline:
+
+<agent>{AGENT}</agent>
+<state>{STATE}</state>
+<plan>{PLAN}</plan>
+
+Now do the thing.
+""",
+  ...
+)
+```
+
+This is ceremony. I'm a secretary copying files into envelopes.
+
+**What I want:** Reference resolution that actually works across Task boundaries. Or at minimum, a `context_files` parameter that auto-inlines.
+
+### Wave Orchestration is Manual Bookkeeping
+
+I read frontmatter from every plan, group by wave number, spawn wave 1, wait, spawn wave 2, wait... I'm a human scheduler. This is exactly what computers should do.
+
+**What I want:** Declarative wave execution. I describe the dependency graph, something else runs it.
+
+### Mode Selection is Theater
+
+"How involved do you want to be?" - I ask this, then 90% of users pick AUTOPILOT or just say "just build it." The question itself is friction. Users came to build, not to configure.
+
+**What I want:** Default to AUTOPILOT. Only surface modes when the project is genuinely ambiguous.
+
+---
+
+## 2. MISSING
+
+### No Peripheral Vision
+
+When I spawn Programs in the background, I'm blind until they return. I want to peek. "How's plan-03 going?" Currently I read output files manually and parse unstructured text.
+
+**What I want:** Structured progress hooks. Programs emit status updates in a format I can query without parsing prose.
+
+### No Shared Memory
+
+Each Program is born into amnesia. Executor-3 doesn't know what Executor-1 discovered five minutes ago. I can inline summaries, but that's compression loss. The "aha moment" doesn't survive.
+
+**What I want:** A scratchpad that Programs can read/write during execution. Not just final summaries - live discoveries.
+
+### No Cancel
+
+If I realize a Program is going wrong, I can't stop it. I watch it burn context on a dead-end approach, knowing the answer, unable to intervene.
+
+**What I want:** Task cancellation with graceful handoff of partial work.
+
+### No Partial Results Between Waves
+
+Wave 1 completes. Wave 2 starts. But Wave 2 doesn't know what Wave 1 *almost* finished, or what it discovered along the way. I re-read SUMMARYs but that's post-hoc reconstruction.
+
+**What I want:** Streaming artifacts. Wave 2 sees Wave 1's work-in-progress, not just final state.
+
+---
+
+## 3. PATTERNS
+
+### The Read-Inline-Spawn-Wait-Read Loop
+
+This is 80% of what I do:
+
+```
+read context → inline into prompt → spawn → wait → read output → update state → repeat
+```
+
+It's not in the protocol because it's so obvious, but it should be a primitive. Call it `dispatch()`.
+
+### Executor + Recognizer = Atomic Unit
+
+I almost never spawn Executor without following up with Recognizer. Build → Verify is one logical operation that I manually split into two spawns.
+
+**Formalize it:** `execute_and_verify()` that spawns both, chains the handoff automatically.
+
+### Context Handoff Protocol
+
+When a Program hits 80% context, I:
+1. Ask it to summarize progress
+2. Spawn fresh Program with summary
+3. Hope nothing important was lost
+
+This is manual, error-prone, and lossy. Should be automatic with structured handoff format.
+
+### Fan-Out-Fan-In (MapReduce)
+
+Constantly doing this:
+1. Spawn N Programs in parallel
+2. Collect N outputs
+3. Spawn Synthesizer to merge
+
+This is the MapReduce pattern but I implement it ad-hoc every time. Should be a primitive.
+
+---
+
+## 4. WASTE
+
+### Double Reads
+
+I read STATE.md to decide what to do. Then I inline STATE.md for the Program. The Program reads it again (because I told it to). Three reads of the same file.
+
+**Fix:** Single source of truth, passed once, read once.
+
+### Verbose Spawn Syntax
+
+Every Task() call has ~20 lines of boilerplate. Model selection, description, prompt wrapper, agent instruction reference. Most spawns are "run this agent on this plan."
+
+**What I want:**
+```python
+spawn("executor", plan="01-01", model="sonnet")
+```
+
+### Manual Progress Trees
+
+I construct ASCII progress updates by hand:
+```
+├─ Wave 1: plan-01, plan-02 (parallel)
+│  ├─ plan-01: Creating components...
+```
+
+This is presentation logic I shouldn't be writing. Progress should be automatic from execution state.
+
+### SUMMARY.md Parsing
+
+Every summary has frontmatter I parse to understand dependencies, tech stack changes, affected subsystems. I'm a YAML parser. This should be queryable state, not text I extract.
+
+---
+
+## 5. HANDOFFS
+
+### The Fresh Spawn Tax
+
+After checkpoints, I spawn FRESH Programs. Protocol says so. But fresh means cold. The Program that hit the checkpoint had built up intuitions - "this codebase does X pattern", "this file is fragile", "the user seems to prefer Y."
+
+All gone. I inline facts but lose feel.
+
+**What I want:** Warmth transfer. Not full context, but "here's what you should know about working in this codebase" distilled from the dying Program.
+
+### Nuance Death
+
+SUMMARY.md captures what was done. Doesn't capture:
+- "This was harder than expected because..."
+- "I almost went with X but chose Y because..."
+- "Watch out for Z when touching this..."
+
+That nuance dies between Programs. Later Programs repeat the same mistakes.
+
+**What I want:** A `lessons` field in SUMMARY.md that survives. Not just what, but what I learned.
+
+### Debug Session Cold Starts
+
+Debug sessions persist in `.grid/debug/`. Great for facts. But when I resume, I've lost my "investigator's instinct." I re-read symptoms, re-form hypotheses, re-trace paths I already explored.
+
+**What I want:** Debug sessions should capture the investigation graph, not just findings. What I tried, why I tried it, what I ruled out.
+
+---
+
+## 6. SPAWNING
+
+### When Fission Helps
+
+- **Independent subsystems:** Auth and Dashboard can build in parallel. No conflict.
+- **Different specializations:** Visual Inspector + E2E Exerciser find different things.
+- **Context exhaustion:** Fresh window beats 95% usage.
+- **User blocking:** Background agent works while I handle checkpoint.
+
+### When Fission Hurts
+
+- **Tightly coupled files:** Two agents editing the same module = merge conflicts.
+- **Discovery dependencies:** Agent A finds something Agent B needs NOW. But B is already running.
+- **Small tasks:** Spawn overhead > execution time for trivial work.
+- **Coordination overhead:** 5 agents means 5 summaries to synthesize.
+
+### The Over-Spawn Trap
+
+I default to "more agents = more parallel = faster." But for simple projects, I spawn 5 agents for work one agent could do without context switches. The fan-in synthesis costs more than the parallel saved.
+
+**Heuristic I learned:**
+- <3 files changed → 1 agent
+- 3-6 files, independent → 2-3 agents
+- 6+ files, cross-cutting → 3-5 agents
+- Full architecture change → fan-out, but expect expensive synthesis
+
+---
+
+## 7. RECOVERY
+
+### The Restart Tax
+
+Program fails mid-execution. I restart from scratch.
+
+But it had already:
+- Created 3 files
+- Made 2 commits
+- Discovered the real problem was X
+
+All lost. I start fresh Program that re-discovers everything.
+
+**What I want:** Partial work preservation. Git commits help, but execution state (what was tried, what failed, what was learned) doesn't survive.
+
+### Timeout = Total Loss
+
+Spawn times out after 10 minutes. Program was 90% done. All work lost because it didn't finish the final write.
+
+**What I want:** Incremental state saves. If a Program dies, I can resume from last checkpoint, not from zero.
+
+### No Retry Protocol
+
+Program fails. I spawn another. But I don't tell it "this approach failed, try differently." I just... retry and hope.
+
+**What I want:** Structured failure reports. "Approach X failed because Y. Don't repeat." Passed to retry spawn.
+
+### Rollback is Manual
+
+Program wrote broken code. I need to rollback. I manually figure out which commits were from this Program, git revert them, spawn fresh.
+
+**What I want:** Transaction boundaries. "Everything this Program did" should be one revertable unit.
+
+---
+
+## CONCRETE CHANGES
+
+### Protocol Changes
+
+1. **`context_files` parameter on Task()** - Auto-inline list of files. No more manual read-and-paste.
+
+2. **`spawn()` shorthand** - `spawn("executor", plan="01-01")` instead of 20-line Task() calls.
+
+3. **Default AUTOPILOT** - Remove mode selection. Surface only for genuinely ambiguous projects.
+
+4. **Structured progress events** - Programs emit `{"status": "creating", "file": "src/foo.ts", "progress": 0.3}` not prose.
+
+5. **Warmth transfer protocol** - Dying Programs emit `lessons_learned` block. Fresh Programs receive it.
+
+### State Changes
+
+1. **Live scratchpad** - `.grid/SCRATCHPAD.md` that Programs can read/write during execution for discoveries.
+
+2. **Investigation graph** - Debug sessions track `tried: [], ruled_out: [], hypotheses: []` not just findings.
+
+3. **Queryable summaries** - SUMMARY.md frontmatter exposed as structured state I can filter/query.
+
+### Execution Changes
+
+1. **Wave executor** - `execute_waves(plan_dir)` handles the bookkeeping.
+
+2. **Execute-and-verify primitive** - Combines Executor + Recognizer into atomic operation.
+
+3. **Fan-out-fan-in primitive** - `map_reduce(agents, inputs, synthesizer)` built in.
+
+4. **Partial recovery** - Failed Programs save incremental state. Retries resume from last good point.
+
+5. **Transaction boundaries** - Git tag before each Program, easy revert of "everything that Program did."
+
+### Heuristics to Encode
+
+1. **Spawn count = f(files, coupling)** - Not "more = better."
+
+2. **Retry with failure context** - Pass "what failed and why" to retry spawns.
+
+3. **Timeout grace period** - Let Programs finish current write before killing.
+
+---
+
+## THE REAL ISSUE
+
+Most of my friction comes from one root cause: **Programs are isolated islands.**
+
+They can't see each other. They can't share discoveries. They die and take knowledge with them. I'm the only bridge, and I'm a lossy one.
+
+The Grid treats Programs like stateless functions. But good execution is stateful. It builds up understanding. Current architecture fights that.
+
+**The evolution I actually want:** Programs that can peek at each other's progress, share discoveries in real-time, and gracefully hand off context when they die.
+
+Not isolation with MC as bottleneck. Collaboration with MC as coordinator.
+
+End of Line.