the-grid-cc 1.7.13 → 1.7.15

package/docs/RESEARCH_CONFIG.md

@@ -0,0 +1,511 @@
+# Research-First Configuration
+
+**Technical Reference**
+**Version:** 1.0
+**Date:** 2026-01-23
+
+---
+
+## Overview
+
+This document specifies configuration options for The Grid's Research-First Architecture. Research configuration controls when and how Scout and Researcher programs are spawned before planning.
+
+---
+
+## Configuration Schema
+
+### Proposed `.grid/config.json`
+
+```json
+{
+  "version": "1.0",
+  "project": {
+    "name": "my-project",
+    "cluster": "project-cluster"
+  },
+  "research": {
+    "enabled": true,
+    "scout": {
+      "enabled": true,
+      "timeout_seconds": 120,
+      "max_files_analyzed": 1000,
+      "depth_limit": 4,
+      "skip_dirs": ["node_modules", ".git", "dist", "build", ".next"]
+    },
+    "researcher": {
+      "enabled": true,
+      "timeout_seconds": 300,
+      "max_researchers_parallel": 3,
+      "max_queries_per_topic": 10,
+      "search_tool_preference": ["mcp__exa__get_code_context_exa", "WebSearch", "WebFetch"]
+    },
+    "cache": {
+      "enabled": true,
+      "ttl_hours": 24,
+      "max_entries": 50,
+      "max_size_mb": 100,
+      "eviction_policy": "lru"
+    },
+    "skip_conditions": {
+      "quick_mode_eligible": true,
+      "debug_mode": true,
+      "explicit_user_skip": true,
+      "all_tech_cached": true
+    }
+  }
+}
+```
+
+---
+
+## Configuration Fields
+
+### `research.enabled` (boolean, default: `true`)
+
+Master switch for research-first architecture.
+
+- `true`: MC spawns Scout/Researcher before Planner
+- `false`: MC spawns Planner directly (legacy behavior)
+
+### `research.scout.enabled` (boolean, default: `true`)
+
+Enable Scout reconnaissance of existing codebases.
+
+- `true`: Scout analyzes existing code before planning
+- `false`: Skip Scout, no codebase context
+
+### `research.scout.timeout_seconds` (integer, default: `120`)
+
+Maximum time Scout can spend on reconnaissance.
+
+- Recommended: `120` (2 minutes)
+- Range: `60-300`
+
+### `research.scout.max_files_analyzed` (integer, default: `1000`)
+
+Maximum number of files Scout will analyze before early exit.
+
+- Prevents performance issues on massive codebases
+- Scout samples first N files for pattern detection
+
+### `research.scout.depth_limit` (integer, default: `4`)
+
+Maximum directory depth for tree traversal.
+
+- Prevents deep recursion on complex directory structures
+- Most patterns detectable within 4 levels
+
+### `research.scout.skip_dirs` (array[string], default: `["node_modules", ".git", "dist", "build", ".next"]`)
+
+Directories Scout will never enter.
+
+- Always respect `.gitignore` patterns
+- Add project-specific build directories
+
+### `research.researcher.enabled` (boolean, default: `true`)
+
+Enable Researcher web intelligence gathering.
+
+- `true`: MC spawns Researchers for external context
+- `false`: Skip research, plan with training knowledge only
+
+### `research.researcher.timeout_seconds` (integer, default: `300`)
+
+Maximum time per Researcher Program.
+
+- Recommended: `300` (5 minutes)
+- Range: `180-600`
+
+### `research.researcher.max_researchers_parallel` (integer, default: `3`)
+
+Maximum number of Researcher Programs spawned in parallel.
+
+- Balances speed vs. API rate limits
+- Each Researcher gets fresh 200k context
+
+### `research.researcher.max_queries_per_topic` (integer, default: `10`)
+
+Maximum search queries per research topic.
+
+- Prevents runaway searches
+- Ensures timely completion
+
+### `research.researcher.search_tool_preference` (array[string], default: `["mcp__exa__get_code_context_exa", "WebSearch", "WebFetch"]`)
+
+Ordered list of preferred search tools.
+
+Researcher tries tools in order based on query type:
+- Code examples: `mcp__exa__get_code_context_exa`
+- Best practices: `WebSearch`
+- Documentation: `WebFetch` with URL
+
+### `research.cache.enabled` (boolean, default: `true`)
+
+Enable research caching.
+
+- `true`: Cache research outputs, check cache before searching
+- `false`: Always fresh research (slow)
+
+### `research.cache.ttl_hours` (integer, default: `24`)
+
+Time-to-live for cache entries in hours.
+
+- Recommended: `24` (balance freshness/speed)
+- Range: `1-168` (1 hour to 1 week)
+
+### `research.cache.max_entries` (integer, default: `50`)
+
+Maximum cached research topics.
+
+- Prevents unbounded cache growth
+- LRU eviction when limit reached
+
+### `research.cache.max_size_mb` (integer, default: `100`)
+
+Maximum total cache size in megabytes.
+
+- Hard limit to prevent disk bloat
+- Evicts oldest entries when exceeded
+
+### `research.cache.eviction_policy` (string, default: `"lru"`)
+
+Cache eviction strategy.
+
+Options:
+- `"lru"`: Least Recently Used
+- `"fifo"`: First In First Out
+- `"ttl"`: Time-based only
+
+### `research.skip_conditions.*` (boolean, all default: `true`)
+
+Conditions under which research phase is skipped.
+
+| Condition | When It Triggers |
+|-----------|-----------------|
+| `quick_mode_eligible` | User request is simple (≤5 files affected) |
+| `debug_mode` | User says "debug" or "fix bug" |
+| `explicit_user_skip` | User says "skip research" |
+| `all_tech_cached` | All detected technologies already cached |
+
+---
+
+## Runtime Configuration
+
+### Environment Variables
+
+Research behavior can be overridden via environment variables:
+
+```bash
+# Disable research entirely
+export GRID_RESEARCH_ENABLED=false
+
+# Force fresh research (ignore cache)
+export GRID_RESEARCH_FORCE_FRESH=true
+
+# Increase researcher timeout
+export GRID_RESEARCHER_TIMEOUT=600
+
+# Limit parallel researchers
+export GRID_MAX_RESEARCHERS=2
+```
+
+### User Commands
+
+Users can control research via natural language:
+
+| User Says | Effect |
+|-----------|--------|
+| "skip research" | Skip research phase for this request |
+| "fresh research" | Ignore cache, research anew |
+| "research only" | Run research, don't plan/execute |
+| "deep research" | Extended timeout (10 min vs 5 min) |
+| "quick mode" | Skip research, use /quick workflow |
+
+---
+
+## Configuration Discovery
+
+### Loading Priority
+
+MC loads configuration in this order (first found wins):
+
+1. `.grid/config.json` - Project-specific config
+2. `~/.claude/grid-config.json` - User global config
+3. Built-in defaults (as documented above)
+
+### Configuration Validation
+
+On load, MC validates:
+- All boolean fields are `true` or `false`
+- All integer fields are within valid ranges
+- `skip_dirs` contains at least `[".git", "node_modules"]`
+- `search_tool_preference` contains valid tool names
+
+Invalid config falls back to defaults with warning.
+
+---
+
+## Integration with MC
+
+### Research Phase Decision Flow
+
+```python
+def should_run_research(user_request: str, config: dict) -> dict:
+    """Determine if research phase should run."""
+
+    if not config['research']['enabled']:
+        return {'run': False, 'reason': 'Research disabled in config'}
+
+    # Check skip conditions
+    if config['research']['skip_conditions']['explicit_user_skip']:
+        if 'skip research' in user_request.lower():
+            return {'run': False, 'reason': 'User explicitly skipped'}
+
+    if config['research']['skip_conditions']['quick_mode_eligible']:
+        if quick_mode_eligible(user_request):
+            return {'run': False, 'reason': 'Quick mode eligible'}
+
+    if config['research']['skip_conditions']['debug_mode']:
+        if is_debug_request(user_request):
+            return {'run': False, 'reason': 'Debug mode'}
+
+    # Check if all tech is cached
+    if config['research']['skip_conditions']['all_tech_cached']:
+        techs = extract_technologies(user_request)
+        if all(is_cached(tech) for tech in techs):
+            return {'run': False, 'reason': 'All tech cached'}
+
+    return {'run': True, 'reason': 'Research needed'}
+```
+
+### MC Spawn Logic
+
+```python
+def research_phase(user_request: str, config: dict) -> dict:
+    """Execute research phase with configuration."""
+
+    decision = should_run_research(user_request, config)
+    if not decision['run']:
+        return {'skipped': True, 'reason': decision['reason']}
+
+    context = {}
+
+    # Scout Phase
+    if config['research']['scout']['enabled'] and has_existing_code():
+        scout_config = config['research']['scout']
+        scout_prompt = f"""
+First, read ~/.claude/agents/grid-scout.md for your role.
+
+<config>
+timeout: {scout_config['timeout_seconds']}s
+max_files: {scout_config['max_files_analyzed']}
+depth_limit: {scout_config['depth_limit']}
+skip_dirs: {scout_config['skip_dirs']}
+</config>
+
+Scan the codebase at {os.getcwd()}.
+Return structured scout report.
+"""
+        context['codebase'] = Task(
+            prompt=scout_prompt,
+            subagent_type="general-purpose",
+            description="Scout codebase reconnaissance"
+        )
+
+    # Researcher Phase
+    if config['research']['researcher']['enabled']:
+        research_needs = extract_research_needs(user_request)
+
+        # Check cache
+        cache_config = config['research']['cache']
+        if cache_config['enabled']:
+            cached, needed = check_research_cache(research_needs, cache_config)
+            context['cached_research'] = cached
+        else:
+            needed = research_needs
+
+        # Spawn researchers (parallel)
+        researcher_config = config['research']['researcher']
+        max_parallel = researcher_config['max_researchers_parallel']
+        needed = needed[:max_parallel]  # Limit spawns
+
+        research_tasks = []
+        for need in needed:
+            researcher_prompt = f"""
+First, read ~/.claude/agents/grid-researcher.md for your role.
+
+<config>
+timeout: {researcher_config['timeout_seconds']}s
+max_queries: {researcher_config['max_queries_per_topic']}
+tools: {researcher_config['search_tool_preference']}
+</config>
+
+<research_request>
+{need}
+</research_request>
+
+Research and return structured context.
+"""
+            task = Task(
+                prompt=researcher_prompt,
+                subagent_type="general-purpose",
+                description=f"Research {need['topic']}"
+            )
+            research_tasks.append(task)
+
+        context['fresh_research'] = research_tasks  # Spawned in parallel
+
+    return context
+```
+
+---
+
+## Migration Path
+
+### Phase 1: Infrastructure Only
+- Create directories: `.grid/research/`, `.grid/research_cache/`, `.grid/scout/`
+- Create README files documenting structure
+- Initialize `index.json` cache index
+
+### Phase 2: Agent Validation
+- Verify `grid-scout.md` and `grid-researcher.md` are properly formatted
+- Test spawning agents with mock config
+
+### Phase 3: MC Integration
+- Add config loading to MC initialization
+- Implement research phase decision logic
+- Add research context to Planner prompts
+
+### Phase 4: User Experience
+- Add user command parsing ("skip research", "fresh research")
+- Document research phase in `/grid:help`
+- Add research metrics to progress output
+
+---
+
+## Example Configuration Files
+
+### Minimal Config (Defaults)
+
+```json
+{
+  "version": "1.0",
+  "research": {
+    "enabled": true
+  }
+}
+```
+
+All other settings use built-in defaults.
+
+### Research Disabled
+
+```json
+{
+  "version": "1.0",
+  "research": {
+    "enabled": false
+  }
+}
+```
+
+Legacy behavior - no Scout or Researcher, plan directly.
+
+### Aggressive Research
+
+```json
+{
+  "version": "1.0",
+  "research": {
+    "enabled": true,
+    "researcher": {
+      "max_researchers_parallel": 5,
+      "timeout_seconds": 600
+    },
+    "cache": {
+      "ttl_hours": 1,
+      "max_entries": 100
+    },
+    "skip_conditions": {
+      "quick_mode_eligible": false,
+      "debug_mode": false,
+      "all_tech_cached": false
+    }
+  }
+}
+```
+
+Research on every request, fresh results, extended timeouts.
+
+### Large Codebase Optimized
+
+```json
+{
+  "version": "1.0",
+  "research": {
+    "scout": {
+      "timeout_seconds": 180,
+      "max_files_analyzed": 5000,
+      "depth_limit": 6,
+      "skip_dirs": [
+        "node_modules", ".git", "dist", "build",
+        "vendor", "target", ".next", ".nuxt"
+      ]
+    }
+  }
+}
+```
+
+Longer Scout timeout, more files, deeper traversal for monorepos.
+
+---
+
+## Observability
+
+### Research Metrics Logged
+
+```yaml
+research_metrics:
+  scout:
+    duration_ms: 1234
+    files_scanned: 456
+    patterns_detected: 8
+    constraints_found: 12
+
+  researcher:
+    topics_researched: 3
+    queries_executed: 18
+    cache_hits: 2
+    cache_misses: 1
+    avg_query_time_ms: 2345
+
+  cache:
+    hit_rate: 0.67
+    entries_cached: 15
+    size_mb: 12.3
+    evictions: 2
+```
+
+### Log Files
+
+```
+.grid/logs/research/
+├── scout_{timestamp}.log
+└── researcher_{topic}_{timestamp}.log
+```
+
+---
+
+## Summary
+
+Research configuration provides fine-grained control over The Grid's research-first architecture:
+
+1. **`research.enabled`** - Master switch
+2. **Scout settings** - Control codebase reconnaissance
+3. **Researcher settings** - Control web intelligence gathering
+4. **Cache settings** - Control research reuse
+5. **Skip conditions** - When to bypass research
+
+Configuration is optional - sensible defaults enable research-first behavior out of the box.
+
+**End of Line.**