the-grid-cc 1.7.12 → 1.7.14

package/docs/RESEARCH_FIRST.md

@@ -0,0 +1,591 @@
+# Research-First Architecture
+
+**Technical Design Document**
+**Version:** 1.0
+**Author:** Program 4 (Research Swarm Executor)
+**Date:** 2026-01-23
+
+---
+
+## Executive Summary
+
+This document describes the **Research-First Architecture** for The Grid - a mandatory research swarm that runs BEFORE any planning phase. The architecture introduces two new agent types (Researcher and Scout) that gather context from external sources and existing codebases, enabling better-informed planning and execution.
+
+### Key Benefits
+- **Informed Planning:** Planner receives structured context about best practices, patterns, and constraints
+- **Reduced Rework:** Discovering constraints BEFORE building prevents costly pivots
+- **Current Knowledge:** Web search brings 2024-2025 best practices beyond training cutoff
+- **Faster Execution:** Cached research eliminates repeated searches
+- **Parallel Intelligence:** Multiple researchers can gather context simultaneously
+
+---
+
+## Architecture Overview
+
+### Current Flow (Before)
+```
+User Request → Planner → Executor → Recognizer
+```
+
+### New Flow (Research-First)
+```
+User Request → Scout → Researcher(s) → Planner → Executor → Recognizer
+                ↓           ↓
+           Codebase     External
+            Context      Context
+```
+
+### Flow Decision Tree
+```
+                    User Request
+                         │
+                         ▼
+              ┌─────────────────────┐
+              │  Quick Mode Check   │
+              └─────────────────────┘
+                    │         │
+            ≤5 files│         │ >5 files
+                    ▼         ▼
+              ┌─────────┐  ┌─────────────────┐
+              │ /quick  │  │ Research Phase  │
+              └─────────┘  └─────────────────┘
+                                  │
+                    ┌─────────────┴─────────────┐
+                    ▼                           ▼
+              ┌─────────┐                 ┌───────────┐
+              │  Scout  │                 │ Researcher│
+              │(codebase)│                │ (external)│
+              └─────────┘                 └───────────┘
+                    │                           │
+                    └─────────────┬─────────────┘
+                                  ▼
+                         ┌───────────────┐
+                         │ Context Cache │
+                         └───────────────┘
+                                  │
+                                  ▼
+                         ┌───────────────┐
+                         │    Planner    │
+                         │(with context) │
+                         └───────────────┘
+                                  │
+                                  ▼
+                         ┌───────────────┐
+                         │   Executor    │
+                         └───────────────┘
+```
+
+---
+
+## Component Specifications
+
+### 1. Scout Agent (`grid-scout.md`)
+
+**Purpose:** Rapid reconnaissance of existing codebase
+**Time Budget:** < 2 minutes
+**Output:** `.grid/scout/RECON_{timestamp}.md`
+
+#### Capabilities
+| Capability | Method | Time |
+|------------|--------|------|
+| Structure Scan | `find`, `glob` | 30s |
+| Technology Detection | Package file analysis | 30s |
+| Pattern Detection | Grep + file existence | 30s |
+| Constraint Discovery | Schema + API analysis | 30s |
+
+#### Key Outputs
+```yaml
+scout_output:
+  technology_stack:
+    languages: [TypeScript, Python]
+    frameworks: [Next.js 14, FastAPI]
+    databases: [PostgreSQL via Prisma]
+
+  detected_patterns:
+    - name: "App Router"
+      confidence: HIGH
+    - name: "Server Components"
+      confidence: MEDIUM
+
+  constraints:
+    locked_dependencies: [react@18.2.0]
+    existing_schemas: [prisma/schema.prisma]
+    api_contracts: [src/app/api/*/route.ts]
+
+  conventions:
+    naming: "camelCase for files, PascalCase for components"
+    imports: "absolute imports from @/"
+```
+
+#### Speed Optimizations
+- Parallel file operations
+- Early exit on pattern detection
+- Depth limits on tree traversal
+- Sample-based analysis (first N, not all)
+
+---
+
+### 2. Researcher Agent (`grid-researcher.md`)
+
+**Purpose:** Gather external intelligence from web sources
+**Time Budget:** 5-10 minutes (parallelizable)
+**Output:** `.grid/research_cache/{topic}.md`
+
+#### Mission Types
+| Mission | Triggers | Output |
+|---------|----------|--------|
+| `technology` | New framework/library | Best practices, gotchas |
+| `patterns` | Architecture decisions | Implementation guides |
+| `similar_projects` | Novel project type | Reference architectures |
+| `api_docs` | External integrations | API context, examples |
+
+#### Search Tool Selection
+| Need | Tool | Rationale |
+|------|------|-----------|
+| Best practices | `WebSearch` | Broad coverage |
+| Code examples | `mcp__exa__get_code_context_exa` | Code-optimized |
+| Company info | `mcp__exa__company_research_exa` | Structured data |
+| Documentation | `WebFetch` | Direct access |
+| GitHub repos | `gh` CLI | Precise queries |
+
+#### Query Generation
+```python
+def generate_queries(request: dict) -> list[str]:
+    queries = []
+
+    # Technology queries (3 per tech)
+    for tech in request.get('technologies', []):
+        queries.extend([
+            f"{tech} best practices 2024 2025",
+            f"{tech} production patterns",
+            f"{tech} common mistakes"
+        ])
+
+    # Pattern queries (3 per pattern)
+    for pattern in request.get('patterns', []):
+        queries.extend([
+            f"{pattern} implementation guide",
+            f"{pattern} real world examples",
+            f"when not to use {pattern}"
+        ])
+
+    return queries
+```
+
+#### Confidence Assessment
+| Level | Criteria |
+|-------|----------|
+| HIGH | 3+ corroborating sources, official docs |
+| MEDIUM | 1-2 sources, reputable blogs |
+| LOW | Single source, old content |
+
+---
+
+### 3. Research Cache
+
+**Location:** `.grid/research_cache/`
+
+#### Structure
+```
+.grid/research_cache/
+├── index.json                    # Cache index
+├── nextjs-14-app-router.md       # Cached research
+├── prisma-best-practices.md
+├── stripe-connect-api.md
+└── archive/                      # Expired entries
+```
+
+#### Index Schema
+```json
+{
+  "version": "1.0",
+  "entries": [
+    {
+      "key": "nextjs-14-app-router",
+      "queries": ["Next.js 14 best practices", "App Router patterns"],
+      "created": "2026-01-23T14:00:00Z",
+      "expires": "2026-01-24T14:00:00Z",
+      "file": "nextjs-14-app-router.md",
+      "hit_count": 3
+    }
+  ]
+}
+```
+
+#### Cache Policy
+| Policy | Value | Rationale |
+|--------|-------|-----------|
+| Default TTL | 24 hours | Balance freshness/speed |
+| Max entries | 50 | Prevent bloat |
+| LRU eviction | On limit | Remove least used |
+| Force refresh | User request | Override cache |
+
+#### Cache Hit Algorithm
+```python
+def check_cache(queries: list[str]) -> Optional[str]:
+    index = load_index()
+
+    for entry in index['entries']:
+        # Check query similarity (Jaccard > 0.7)
+        if jaccard_similarity(entry['queries'], queries) > 0.7:
+            if not is_expired(entry):
+                entry['hit_count'] += 1
+                save_index(index)
+                return read(entry['file'])
+
+    return None  # Cache miss
+```
+
+---
+
+## Integration with Master Control
+
+### MC Research Phase Logic
+
+```python
+def research_phase(user_request: str, project_root: str) -> dict:
+    """Execute research phase before planning."""
+
+    # Check if research should be skipped
+    if should_skip_research(user_request):
+        return {"skipped": True, "reason": "Quick mode eligible"}
+
+    context = {}
+
+    # Phase 1: Scout existing codebase (always, if exists)
+    if has_existing_code(project_root):
+        scout_result = Task(
+            prompt="""
+First, read ~/.claude/agents/grid-scout.md for your role.
+
+Scan the codebase at {project_root}.
+Return structured scout report.
+""",
+            subagent_type="general-purpose",
+            description="Scout codebase"
+        )
+        context['codebase'] = scout_result
+
+    # Phase 2: Extract research needs
+    research_needs = extract_research_needs(user_request, context.get('codebase'))
+
+    # Phase 3: Check cache for existing research
+    cached, needed = check_research_cache(research_needs)
+    context['cached_research'] = cached
+
+    # Phase 4: Spawn researchers for uncached needs (parallel)
+    if needed:
+        research_tasks = []
+        for need in needed:
+            task = Task(
+                prompt=f"""
+First, read ~/.claude/agents/grid-researcher.md for your role.
+
+<research_request>
+{need}
+</research_request>
+
+Research and return structured context.
+""",
+                subagent_type="general-purpose",
+                description=f"Research {need['topic']}"
+            )
+            research_tasks.append(task)
+
+        # All tasks execute in parallel
+        context['fresh_research'] = await_all(research_tasks)
+
+    return context
+```
+
+### Skip Conditions
+
+Research is skipped when:
+```python
+def should_skip_research(request: str) -> bool:
+    """Determine if research phase should be skipped."""
+
+    # Quick mode eligible = skip research
+    if quick_mode_eligible(request):
+        return True
+
+    # User explicitly skipped
+    if "skip research" in request.lower():
+        return True
+
+    # Known technology fully cached
+    if all_tech_cached(extract_technologies(request)):
+        return True
+
+    # Debug/fix mode
+    if is_debug_request(request):
+        return True
+
+    return False
+```
+
+---
+
+## Planner Integration
+
+### Planner Prompt Enhancement
+
+```python
+def spawn_planner_with_research(user_request: str, research_context: dict):
+    """Spawn Planner with research context."""
+
+    prompt = f"""
+First, read ~/.claude/agents/grid-planner.md for your role.
+
+<codebase_context>
+{research_context.get('codebase', 'Greenfield project - no existing code')}
+</codebase_context>
+
+<research_context>
+{format_research(research_context.get('cached_research', []))}
+{format_research(research_context.get('fresh_research', []))}
+</research_context>
+
+<user_request>
+{user_request}
+</user_request>
+
+Create execution plan informed by:
+1. Codebase constraints (if existing code)
+2. Best practices from research
+3. Anti-patterns to avoid
+4. Recommended technologies
+
+Your plan should respect existing patterns and leverage researched best practices.
+"""
+
+    return Task(
+        prompt=prompt,
+        subagent_type="general-purpose",
+        description="Plan with research context"
+    )
+```
+
+### Research-Informed Planning Rules
+
+Planner applies research context as:
+```yaml
+planning_rules:
+  # Constraints from Scout
+  must_preserve:
+    - existing_api_contracts
+    - database_schema_structure
+    - naming_conventions
+
+  # Patterns from Researcher
+  should_use:
+    - recommended_patterns
+    - best_practices_2024_2025
+
+  # Anti-patterns from Researcher
+  must_avoid:
+    - documented_anti_patterns
+    - deprecated_approaches
+
+  # Technology from Researcher
+  prefer:
+    - libraries_with_good_research
+    - patterns_with_code_examples
+```
+
+---
+
+## Performance Considerations
+
+### Time Budgets
+| Phase | Target | Max |
+|-------|--------|-----|
+| Scout | 1 min | 2 min |
+| Cache check | 1 sec | 5 sec |
+| Single researcher | 3 min | 5 min |
+| Parallel researchers | 3 min | 5 min |
+| Total research phase | 4 min | 7 min |
+
+### Parallelization Strategy
+```
+Sequential:
+  Scout (1 min) → Cache Check (1 sec) → [If cache miss] Research Phase
+
+Research Phase (Parallel):
+  ├── Researcher 1: Framework docs (3 min)
+  ├── Researcher 2: Pattern research (3 min)
+  └── Researcher 3: API docs (3 min)
+       └── Total: 3 min (not 9 min)
+```
+
+### Context Budget
+| Agent | Context Budget |
+|-------|---------------|
+| Scout | 10% max |
+| Researcher | 20% per instance |
+| Research output to Planner | 15% max |
+
+Output compression ensures Planner receives concise, actionable context.
+
+---
+
+## Configuration
+
+### `.grid/config.json` Extensions
+```json
+{
+  "research": {
+    "enabled": true,
+    "cache_ttl_hours": 24,
+    "max_researchers": 3,
+    "scout_timeout_seconds": 120,
+    "researcher_timeout_seconds": 300,
+    "skip_for_quick_mode": true
+  }
+}
+```
+
+### User Controls
+| Command | Effect |
+|---------|--------|
+| "skip research" | Bypass research phase |
+| "fresh research" | Ignore cache, research anew |
+| "research only" | Run research, don't plan |
+| "deep research" | Extended time budget |
+
+---
+
+## Error Handling
+
+### Scout Failures
+```yaml
+failure_modes:
+  timeout:
+    action: "Proceed with partial results"
+    fallback: "Manual constraint discovery"
+
+  permission_denied:
+    action: "Report inaccessible areas"
+    fallback: "User provides constraints"
+
+  empty_directory:
+    action: "Report greenfield"
+    fallback: "Standard patterns"
+```
+
+### Researcher Failures
+```yaml
+failure_modes:
+  api_timeout:
+    action: "Retry once, then skip topic"
+    fallback: "Use training knowledge"
+
+  no_results:
+    action: "Report low confidence"
+    fallback: "Mark for manual review"
+
+  rate_limited:
+    action: "Queue for later"
+    fallback: "Proceed without"
+```
+
+---
+
+## Observability
+
+### Research Metrics
+```yaml
+metrics:
+  scout:
+    - scan_duration_ms
+    - files_analyzed
+    - patterns_detected
+    - constraints_found
+
+  researcher:
+    - queries_executed
+    - sources_found
+    - cache_hit_rate
+    - avg_query_time_ms
+
+  overall:
+    - research_phase_duration_ms
+    - cache_hit_rate
+    - context_compression_ratio
+```
+
+### Logging
+```
+.grid/logs/research/
+├── scout_2026-01-23T14:00:00.log
+├── researcher_nextjs_2026-01-23T14:01:00.log
+└── researcher_prisma_2026-01-23T14:01:00.log
+```
+
+---
+
+## Migration Path
+
+### Phase 1: Scout Only (Week 1)
+- Deploy Scout agent
+- Add Scout to MC flow before Planner
+- Measure planning improvement
+
+### Phase 2: Researcher Integration (Week 2)
+- Deploy Researcher agent
+- Add cache infrastructure
+- Integrate with Planner prompts
+
+### Phase 3: Full Integration (Week 3)
+- Parallel researcher spawning
+- Cache optimization
+- Metrics and observability
+
+### Backward Compatibility
+- Research phase can be disabled via config
+- Quick mode bypasses research by default
+- Existing projects continue to work
+
+---
+
+## Security Considerations
+
+### Data Handling
+- Research cache in `.grid/` (gitignored)
+- No secrets in research output
+- External queries sanitized
+
+### Rate Limiting
+- Max 10 queries per researcher
+- 2-second delay between web searches
+- Respect API rate limits
+
+---
+
+## Future Enhancements
+
+### Planned
+1. **Semantic caching:** Match by meaning, not exact query
+2. **Learning loop:** Feed successful patterns back to research
+3. **Team knowledge:** Shared research cache across projects
+4. **Custom sources:** User-defined authoritative sources
+
+### Considered
+- RAG over codebase for deep context
+- ML-based pattern detection
+- Automated research scheduling
+
+---
+
+## Summary
+
+The Research-First Architecture introduces mandatory intelligence gathering before planning:
+
+1. **Scout** rapidly analyzes existing codebases (< 2 min)
+2. **Researcher** gathers external best practices (3-5 min, parallelizable)
+3. **Cache** prevents repeated searches (24h TTL)
+4. **Planner** receives structured context for informed planning
+
+This architecture ensures The Grid builds on current knowledge and respects existing constraints, reducing rework and improving plan quality.
+
+**End of Line.**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "the-grid-cc",
-  "version": "1.7.12",
+  "version": "1.7.14",
   "description": "Agent orchestration for Claude Code. You talk to Master Control. Master Control handles the rest.",
   "main": "index.js",
   "bin": {