the-grid-cc 1.7.14 → 1.7.16

package/docs/RESEARCH_INFRASTRUCTURE.md

@@ -0,0 +1,429 @@
+# Research Infrastructure Setup
+
+**Implementation Complete**
+**Version:** 1.0
+**Date:** 2026-01-23
+
+---
+
+## Overview
+
+The research-first infrastructure is now functional and ready for MC integration. This document describes what was created and how to use it.
+
+---
+
+## Directory Structure Created
+
+```
+.grid/                              # Local state (gitignored)
+├── research/                       # Fresh research outputs
+│   ├── README.md                  # Documentation
+│   ├── .gitignore                 # Ignore *.md except README
+│   └── {topic}-{timestamp}.md     # Research outputs (created by Researchers)
+│
+├── research_cache/                 # Cached research (24h TTL)
+│   ├── README.md                  # Documentation
+│   ├── .gitignore                 # Ignore *.md except README/index
+│   ├── index.json                 # Cache index with metadata
+│   └── {topic-slug}.md            # Cached research (reusable)
+│
+└── scout/                          # Codebase reconnaissance
+    ├── README.md                   # Documentation
+    ├── .gitignore                  # Ignore RECON_*.md except README
+    └── RECON_{timestamp}.md        # Scout reports (created by Scouts)
+```
+
+All directories are in `.grid/` which is gitignored for local state isolation.
+
+---
+
+## Files Created
+
+### Documentation
+
+| File | Purpose |
+|------|---------|
+| `.grid/research/README.md` | Documents research output directory |
+| `.grid/research_cache/README.md` | Documents cache structure and policies |
+| `.grid/scout/README.md` | Documents scout report directory |
+| `docs/RESEARCH_CONFIG.md` | Configuration schema reference |
+| `docs/MC_RESEARCH_INTEGRATION.md` | MC integration implementation guide |
+| `docs/RESEARCH_INFRASTRUCTURE.md` | This file - setup summary |
+
+### Gitignores
+
+| File | Purpose |
+|------|---------|
+| `.grid/research/.gitignore` | Ignore research outputs, keep README |
+| `.grid/research_cache/.gitignore` | Ignore cached files, keep README + index |
+| `.grid/scout/.gitignore` | Ignore scout reports, keep README |
+
+### State Files
+
+| File | Purpose |
+|------|---------|
+| `.grid/research_cache/index.json` | Empty cache index (ready for entries) |
+
+---
+
+## Agent Files Verified
+
+Both agent files are properly formatted for spawning:
+
+### `/Users/jacweath/grid/agents/grid-researcher.md`
+
+**Status:** ✅ Functional
+
+**Contents:**
+- Role definition
+- Mission types (technology, patterns, similar projects, API docs)
+- Research protocol (query generation, parallel search, context assembly)
+- Caching protocol (cache structure, hit logic, invalidation)
+- Search tool selection
+- Output format
+- Quality rules
+- Integration with planning
+
+**Ready to spawn:** Yes - MC can spawn with Task tool
+
+### `/Users/jacweath/grid/agents/grid-scout.md`
+
+**Status:** ✅ Functional
+
+**Contents:**
+- Role definition
+- Reconnaissance protocol (4 phases in 2 minutes)
+- Technology detection
+- Pattern detection
+- Constraint discovery
+- Output format (scout report)
+- Speed optimization
+- Special modes (greenfield, monorepo)
+- Integration with research
+
+**Ready to spawn:** Yes - MC can spawn with Task tool
+
+---
+
+## Cache Index Schema
+
+The cache index is initialized and ready:
+
+```json
+{
+  "version": "1.0",
+  "created": "2026-01-23T20:00:00Z",
+  "updated": "2026-01-23T20:00:00Z",
+  "entries": []
+}
+```
+
+As Researchers cache results, entries will be added:
+
+```json
+{
+  "entries": [
+    {
+      "key": "nextjs-14-app-router",
+      "queries": ["Next.js 14 best practices 2024 2025", ...],
+      "created": "2026-01-23T14:00:00Z",
+      "expires": "2026-01-24T14:00:00Z",
+      "file": "nextjs-14-app-router.md",
+      "hit_count": 0,
+      "size_kb": 15
+    }
+  ]
+}
+```
+
+---
+
+## How MC Spawns Researchers
+
+### 1. Research Decision
+
+MC checks if research should run:
+
+```python
+decision = should_run_research(user_request, config)
+if not decision['run']:
+    # Skip research, go direct to Planner
+```
+
+### 2. Scout Spawn (if existing codebase)
+
+```python
+scout_prompt = """
+First, read ~/.claude/agents/grid-scout.md for your role.
+
+<config>
+project_root: /path/to/project
+timeout_seconds: 120
+max_files_analyzed: 1000
+</config>
+
+Scan the codebase. Output to .grid/scout/RECON_{timestamp}.md
+"""
+
+scout_result = Task(
+    prompt=scout_prompt,
+    subagent_type="general-purpose",
+    description="Scout: Codebase reconnaissance"
+)
+```
+
+### 3. Research Needs Extraction
+
+```python
+needs = extract_research_needs(user_request, scout_result)
+# Returns: [
+#   {'type': 'technology', 'topic': 'Next.js 14', 'queries': [...]},
+#   {'type': 'pattern', 'topic': 'Server Components', 'queries': [...]},
+# ]
+```
+
+### 4. Cache Check
+
+```python
+cached, needed = check_research_cache(needs, config)
+# Returns:
+# - cached: Research already in .grid/research_cache/
+# - needed: Topics not cached or expired
+```
+
+### 5. Researcher Spawn (parallel)
+
+```python
+for need in needed:
+    researcher_prompt = f"""
+First, read ~/.claude/agents/grid-researcher.md for your role.
+
+<research_request>
+mission: {need['type']}
+topic: {need['topic']}
+queries: {need['queries']}
+</research_request>
+
+Research and output to .grid/research_cache/{topic}.md
+"""
+
+    Task(
+        prompt=researcher_prompt,
+        subagent_type="general-purpose",
+        description=f"Research: {need['topic']}"
+    )
+```
+
+All Researchers spawn in parallel (single MC message, multiple Task calls).
+
+### 6. Context Assembly
+
+```python
+context = {
+    'scout': scout_result,
+    'cached_research': cached,
+    'fresh_research': fresh_results
+}
+```
+
+### 7. Planner Spawn
+
+```python
+planner_prompt = f"""
+First, read ~/.claude/agents/grid-planner.md for your role.
+
+<codebase_context>
+{context['scout']}
+</codebase_context>
+
+<research_context>
+{format_research(context['cached_research'])}
+{format_research(context['fresh_research'])}
+</research_context>
+
+<user_request>
+{user_request}
+</user_request>
+
+Create plan that respects codebase constraints and uses best practices.
+"""
+
+Task(
+    prompt=planner_prompt,
+    subagent_type="general-purpose",
+    description="Planner: Research-informed plan"
+)
+```
+
+---
+
+## Configuration Schema Added
+
+### Proposed `.grid/config.json`
+
+```json
+{
+  "version": "1.0",
+  "research": {
+    "enabled": true,
+    "scout": {
+      "enabled": true,
+      "timeout_seconds": 120,
+      "max_files_analyzed": 1000,
+      "depth_limit": 4
+    },
+    "researcher": {
+      "enabled": true,
+      "timeout_seconds": 300,
+      "max_researchers_parallel": 3,
+      "max_queries_per_topic": 10
+    },
+    "cache": {
+      "enabled": true,
+      "ttl_hours": 24,
+      "max_entries": 50
+    },
+    "skip_conditions": {
+      "quick_mode_eligible": true,
+      "debug_mode": true,
+      "all_tech_cached": true
+    }
+  }
+}
+```
+
+Full schema documented in: `docs/RESEARCH_CONFIG.md`
+
+---
+
+## Integration Checklist
+
+Research infrastructure is complete. Next steps for full integration:
+
+### Phase 1: Validation (Complete)
+- [x] Create `.grid/research/` directory
+- [x] Create `.grid/research_cache/` directory
+- [x] Create `.grid/scout/` directory
+- [x] Write README files documenting structure
+- [x] Initialize cache index JSON
+- [x] Add .gitignore files
+- [x] Verify agent files are spawn-ready
+- [x] Document configuration schema
+- [x] Document MC integration flow
+
+### Phase 2: MC Integration (Ready to implement)
+- [ ] Add config loading to MC initialization
+- [ ] Implement `should_run_research()` logic
+- [ ] Implement `spawn_scout()` function
+- [ ] Implement `extract_research_needs()` function
+- [ ] Implement `check_research_cache()` function
+- [ ] Implement `spawn_researchers()` function
+- [ ] Implement context assembly
+- [ ] Update Planner spawn with research context
+
+### Phase 3: User Experience (After MC integration)
+- [ ] Add user commands ("skip research", "fresh research")
+- [ ] Add progress output ("Research phase: Scout complete...")
+- [ ] Document in `/grid:help`
+- [ ] Add metrics logging
+
+---
+
+## File Locations Reference
+
+### Agent Definitions
+- `/Users/jacweath/grid/agents/grid-scout.md` - Scout agent
+- `/Users/jacweath/grid/agents/grid-researcher.md` - Researcher agent
+- `/Users/jacweath/grid/agents/grid-planner.md` - Planner agent (receives research)
+
+### Documentation
+- `/Users/jacweath/grid/docs/RESEARCH_FIRST.md` - Architecture design
+- `/Users/jacweath/grid/docs/RESEARCH_CONFIG.md` - Configuration reference
+- `/Users/jacweath/grid/docs/MC_RESEARCH_INTEGRATION.md` - MC integration guide
+- `/Users/jacweath/grid/docs/RESEARCH_INFRASTRUCTURE.md` - This file
+
+### State Directories (Local Only - Gitignored)
+- `/Users/jacweath/grid/.grid/research/` - Fresh research outputs
+- `/Users/jacweath/grid/.grid/research_cache/` - Cached research
+- `/Users/jacweath/grid/.grid/scout/` - Scout reports
+
+### Master Control
+- `/Users/jacweath/grid/commands/grid/mc.md` - MC command definition
+
+---
+
+## Testing Research Infrastructure
+
+### Manual Test: Scout Spawn
+
+```python
+# In MC
+scout_prompt = """
+First, read ~/.claude/agents/grid-scout.md for your role.
+
+<config>
+project_root: /Users/jacweath/grid
+timeout_seconds: 120
+</config>
+
+Scan the Grid codebase. Write report to .grid/scout/RECON_test.md
+"""
+
+Task(prompt=scout_prompt, subagent_type="general-purpose", description="Test Scout")
+```
+
+Expected: Scout report in `.grid/scout/RECON_test.md`
+
+### Manual Test: Researcher Spawn
+
+```python
+# In MC
+researcher_prompt = """
+First, read ~/.claude/agents/grid-researcher.md for your role.
+
+<research_request>
+mission: technology
+topic: Next.js 14 App Router
+queries:
+  - Next.js 14 App Router best practices 2024 2025
+  - Server Components production patterns
+</research_request>
+
+Research and write to .grid/research_cache/nextjs-14-app-router.md
+Update cache index at .grid/research_cache/index.json
+"""
+
+Task(prompt=researcher_prompt, subagent_type="general-purpose", description="Test Researcher")
+```
+
+Expected:
+- Research output in `.grid/research_cache/nextjs-14-app-router.md`
+- Cache index updated with new entry
+
+### Manual Test: Cache Hit
+
+```python
+# Second call with same topic should hit cache
+cached, needed = check_research_cache([
+    {'topic': 'Next.js 14 App Router', 'queries': ['Next.js 14 best practices']}
+])
+
+assert len(cached) == 1  # Hit!
+assert len(needed) == 0  # Nothing to research
+```
+
+---
+
+## Summary
+
+Research-first infrastructure is **FUNCTIONAL**:
+
+1. **Directories created** - `.grid/research/`, `.grid/research_cache/`, `.grid/scout/`
+2. **Documentation complete** - READMEs, config schema, integration guide
+3. **Agents validated** - Scout and Researcher are spawn-ready
+4. **Cache initialized** - Empty index ready for entries
+5. **Gitignores added** - State files properly excluded
+
+**Next Step:** MC integration - implement research phase logic in Master Control.
+
+**End of Line.**