@polymorphism-tech/morph-spec 3.2.0 → 4.3.0

package/docs/next-generation/FEATURES.md

@@ -0,0 +1,688 @@
+# MORPH-SPEC v3.0 - Features
+
+> Detailed features: 7 thread types, context optimization, HOPs, workflows, analytics
+
+**Status:** Design Complete ✅
+**Last Updated:** 2026-02-18
+
+---
+
+## Table of Contents
+
+- [Execution Engine: 7 Thread Types](#execution-engine-7-thread-types)
+- [Context Optimization: R&D Framework](#context-optimization-rd-framework)
+- [Meta-Programming Layer: HOPs](#meta-programming-layer-hops)
+- [Observability & Analytics](#observability--analytics)
+- [Workflows Evolution](#workflows-evolution)
+
+---
+
+## Execution Engine: 7 Thread Types
+
+**Source:** thread-based-engineering-analysis.md (Boris Churnney's 10-15 parallel threads)
+
+### Current Limitation ❌
+
+MORPH runs **1 agent at a time** (sequential execution):
+```bash
+morph-spec feature create user-auth
+# Workflow: proposal → design → tasks → implement
+# Only 1 agent active, ~100 tool calls total, ~4 hours
+```
+
+### Solution: Thread Management System ✅
+
+#### Thread Types Overview
+
+| Type | Pattern | Purpose | Status |
+|------|---------|---------|--------|
+| **Base** | prompt → agent → review | Single workflow | ✅ Current |
+| **P-Thread** | 3-5 parallel threads | Scale through concurrency | ❌ NEW |
+| **C-Thread** | Chained phases + checkpoints | Break work into validated chunks | ✅ Current (5 phases) |
+| **F-Thread** | Best-of-3 agents → pick best | Fusion for confidence | ❌ NEW |
+| **B-Thread** | Nested sub-threads | Squad leaders + specialists | ⚠️ Partial (needs orchestration) |
+| **L-Thread** | Hours-long autonomous | Stop hooks for validation loop | ❌ NEW |
+| **Z-Thread** | Zero-touch, auto-approve | Trust levels enable this | ❌ NEW |
+
+#### New CLI Commands
+
+```bash
+# Parallel execution (P-Thread)
+morph-spec tasks run T001,T002,T003 --parallel --max-concurrent=3
+morph-spec agents spawn-squad user-auth --parallel backend,frontend,infra
+
+# Fusion patterns (F-Thread)
+morph-spec agents fuse --prompt "design auth system" --count=3 --strategy=best-of-n
+morph-spec review aggregate user-auth --agents=agent1,agent2,agent3
+
+# Thread management
+morph-spec threads list              # View all active threads
+morph-spec threads status {id}        # Progress monitoring
+morph-spec threads kill {id}          # Stop background thread
+morph-spec threads analytics          # Performance dashboard
+morph-spec threads wait-all {feature} # Block until all complete
+
+# Trust levels (path to Z-Thread)
+morph-spec trust set user-auth high
+morph-spec trust history
+morph-spec trust auto-calculate user-auth
+```
+
+#### State Extensions
+
+**state.json additions:**
+```json
+{
+  "threads": {
+    "thread-001": {
+      "id": "thread-001",
+      "feature": "user-auth",
+      "agent": "dotnet-senior",
+      "type": "parallel",
+      "status": "running",
+      "startedAt": "2026-02-18T10:00:00Z",
+      "metrics": {
+        "toolCalls": 42,
+        "estimatedCompletion": "2026-02-18T10:30:00Z"
+      }
+    }
+  },
+  "features": {
+    "user-auth": {
+      "threadMetrics": {
+        "totalToolCalls": 247,
+        "threadDuration": "2h 15m",
+        "maxParallelThreads": 3,
+        "avgParallelThreads": 2.1,
+        "checkpointPassRate": 1.0
+      },
+      "trustConfig": {
+        "level": "high",
+        "autoApprove": {
+          "design": true,
+          "tasks": false
+        },
+        "history": [
+          { "feature": "product-catalog", "passRate": 1.0 },
+          { "feature": "order-mgmt", "passRate": 0.95 }
+        ]
+      }
+    }
+  }
+}
+```
+
+### Impact
+
+- **Throughput:** 1 feature/day → 3-5 features/day (+200-400%)
+- **Tool calls:** 100/feature → 250/feature (+150%)
+- **Autonomous runtime:** <30min → >4 hours (+700%)
+- **Approval overhead:** 100% manual → 50% auto (-50%)
+
+---
+
+## Context Optimization: R&D Framework
+
+**Source:** context-engineering-analysis.md (30-40% token reduction via Reduce & Delegate)
+
+### Current Waste ❌
+
+- **CLAUDE.md:** 23K tokens (loaded every session)
+- **Session replay:** 180K tokens (no context bundles)
+- **MCP servers:** All loaded (even unused ones)
+- **No token tracking:** Can't measure optimization
+
+### Solution: 12 Techniques Across 4 Levels ✅
+
+#### Level 1: Beginner (Quick Wins - Week 1)
+
+| Technique | Impact | Effort |
+|-----------|--------|--------|
+| **MCP hygiene** | -10K tokens | 1 day |
+| **Context priming** | -22.5K tokens (23K → 500) | 2 days |
+
+**5 Prime Commands:**
+```bash
+/prime-feature   # Feature implementation (selective standards)
+/prime-bug       # Bug fixing context
+/prime-refactor  # Refactoring guidelines
+/prime-design    # Architecture standards
+/prime-infra     # Infrastructure templates
+```
+
+**Example priming file (.morph/context/feature-prime.md):**
+```markdown
+# Feature Implementation Context (500 tokens vs 23K)
+
+## Active Standards
+- backend/dotnet/core.md
+- backend/api/minimal-api.md
+- core/architecture.md
+
+## Active Agents
+- dotnet-senior (backend)
+- ef-modeler (data)
+- api-designer (contracts)
+
+## Templates Available
+- service.cs, repository.cs, contracts.cs
+
+Full standards: Run /load-standard {name}
+```
+
+#### Level 2: Intermediate (Week 2)
+
+| Technique | Impact | Effort |
+|-----------|--------|--------|
+| **Sub-agent delegation** | Isolate context per domain | 3 days |
+| **Core Four tracking** | Context visibility | 2 days |
+
+**Core Four Dashboard:**
+```bash
+morph-spec core-four user-auth
+
+Output:
+📊 Core Four: user-authentication
+Context: 12,450 tokens (23% of max)
+Model:   Sonnet 4.5 (128K context)
+Prompt:  "Implement user authentication with JWT"
+Tools:   Read (23), Write (8), Edit (12), Bash (5)
+```
+
+#### Level 3: Advanced (Week 3)
+
+| Technique | Impact | Effort |
+|-----------|--------|--------|
+| **Context bundles** | -165K tokens on replay (85%) | 5 days |
+| **Background agents** | Out-of-loop work | 3 days |
+
+**Context Bundle:**
+```json
+{
+  "bundleId": "user-auth-checkpoint-2",
+  "timestamp": "2026-02-18T10:30:00Z",
+  "compressedContext": {
+    "decisions": ["ADR-001: JWT tokens", "ADR-002: BCrypt hashing"],
+    "artifacts": ["User.cs", "AuthController.cs", "UserService.cs"],
+    "nextSteps": ["Implement refresh token rotation"]
+  },
+  "fullContextPath": ".morph/bundles/user-auth-checkpoint-2.json",
+  "replayTokens": 15000
+}
+```
+
+Replay: `morph-spec resume user-auth --from-bundle checkpoint-2` (15K tokens vs 180K)
+
+#### Level 4: Agentic (Week 4)
+
+| Technique | Impact | Effort |
+|-----------|--------|--------|
+| **Micro-agents** | Ultra-specialized (5-10x faster) | 5 days |
+| **Context analytics** | Real-time optimization suggestions | 4 days |
+
+**Micro-Agent Example:**
+```bash
+# Instead of: dotnet-senior (knows 48 standards, 100+ patterns)
+# Use: jwt-token-specialist (knows only JWT, 500 tokens context)
+
+morph-spec micro-agent create jwt-token-specialist \
+  --base-agent=dotnet-senior \
+  --standards=backend/authentication/jwt.md \
+  --templates=jwt-service.cs
+```
+
+**Context Analytics:**
+```bash
+morph-spec analytics context user-auth
+
+Output:
+📊 Context Analytics: user-authentication
+
+Current: 18,200 tokens
+Opportunities:
+  ✅ MCP hygiene: -8K (remove unused context7 server)
+  ✅ Switch to micro-agent: -6K (jwt-token-specialist vs dotnet-senior)
+  ✅ Context bundle from checkpoint 1: -12K
+
+Potential: 18,200 → 5,200 tokens (-71%)
+```
+
+### Impact
+
+- **CLAUDE.md:** 23K → 500 tokens (-95%)
+- **Session replay:** 180K → 15K tokens (-92%)
+- **Overall context:** -30-40% reduction
+- **Response speed:** 5-10x faster (micro-agents)
+
+---
+
+## Meta-Programming Layer: HOPs
+
+**Source:** 4-layer-agentic-stack-analysis.md (prompts that take prompts as parameters)
+
+### Current Limitation ❌
+
+Every agent spawn requires manual prompt composition:
+```bash
+# Manual composition (error-prone, inconsistent)
+Task tool → "You are the Backend Squad Leader (dotnet-senior)...
+Your Domain Leaders: ef-modeler, api-designer...
+Your Mission: Implement user auth...
+Standards: coding.md, architecture.md, dotnet10-compatibility.md...
+Tasks: T001, T002, T003..."
+```
+
+### Solution: 12+ Higher-Order Prompts ✅
+
+**HOP = Prompt template that takes prompts/variables as parameters**
+
+#### HOP Categories
+
+**1. Squad Leaders (3 HOPs)**
+```
+framework/templates/meta-prompts/squad-leaders/
+├── squad-leader.md          # Generic squad leader
+├── backend-squad.md         # Backend-specific (dotnet-senior)
+└── frontend-squad.md        # Frontend-specific (blazor/nextjs)
+```
+
+**Example: backend-squad.md**
+```handlebars
+You are the {{SQUAD_NAME}} Squad Leader ({{AGENT_ID}}).
+
+**Your Domain Leaders:**
+{{#each DOMAIN_LEADERS}}
+- {{this.id}}: {{this.specialty}}
+{{/each}}
+
+**Your Mission:** {{MISSION}}
+
+**Feature Spec:**
+{{SPEC_SUMMARY}}
+
+**Standards to Enforce:**
+{{#each STANDARDS}}
+- {{this}}
+{{/each}}
+
+**Your Tasks (from tasks.json):**
+{{#each TASKS}}
+- {{this.id}}: {{this.description}}
+{{/each}}
+
+**Deliverables:**
+{{#each DELIVERABLES}}
+- {{this}}
+{{/each}}
+
+**Constraints:**
+{{#each CONSTRAINTS}}
+- {{this}}
+{{/each}}
+
+Report back when complete with file list + any blockers.
+```
+
+**Usage:**
+```bash
+morph-spec template render meta-prompts/backend-squad \
+  --vars='{
+    "SQUAD_NAME": "Backend",
+    "AGENT_ID": "dotnet-senior",
+    "MISSION": "Implement user authentication API",
+    "DOMAIN_LEADERS": [
+      {"id": "ef-modeler", "specialty": "Database/EF Core"},
+      {"id": "api-designer", "specialty": "REST endpoints"}
+    ],
+    "STANDARDS": ["coding.md", "architecture.md"],
+    "TASKS": [...],
+    "DELIVERABLES": ["User.cs", "AuthController.cs"],
+    "CONSTRAINTS": ["Use IDbContextFactory", "BCrypt for passwords"]
+  }'
+```
+
+**2. Parallel Workers (2 HOPs)**
+```
+framework/templates/meta-prompts/parallel-workers/
+├── parallel-worker.md       # Generic parallel task
+└── parallel-coordinator.md  # Coordinates N parallel workers
+```
+
+**3. HOPs (Wrapper Patterns) (3 HOPs)**
+```
+framework/templates/meta-prompts/hops/
+├── hop-wrapper.md           # Generic HOP wrapper
+├── hop-retry.md             # Retry logic wrapper
+└── hop-validation.md        # Validation wrapper
+```
+
+**4. Validators (2 HOPs)**
+```
+framework/templates/meta-prompts/validators/
+├── checkpoint-validator.md  # Checkpoint validation
+└── pre-commit-validator.md  # Pre-commit hook validation
+```
+
+**5. Fusion Patterns (2 HOPs)**
+```
+framework/templates/meta-prompts/fusion/
+├── fusion-agent.md          # Agent in fusion pool
+└── fusion-aggregator.md     # Aggregates N results
+```
+
+### Impact
+
+- **Consistency:** All squad spawns use same pattern
+- **Reusability:** 12+ templates vs manual composition every time
+- **Composability:** HOPs can wrap other HOPs (retry → validation → worker)
+- **Error reduction:** Templates validated, versioned, tested
+
+---
+
+## Observability & Analytics
+
+**Source:** thread-based-engineering-analysis.md + context-engineering-analysis.md
+
+### Current Blindness ❌
+
+No visibility into:
+- How many tool calls per feature?
+- How long do features take?
+- Which agents are most effective?
+- Is parallelization working?
+- Are checkpoints passing?
+- Token usage trends?
+
+### Solution: Comprehensive Analytics System ✅
+
+#### 1. Thread Analytics Dashboard
+
+```bash
+morph-spec analytics feature user-auth
+
+Output:
+📊 Thread Analytics: user-authentication
+
+🔧 Tool Calls: 247 (avg: 82/checkpoint)
+⏱️  Duration: 2h 15m
+✅ Checkpoint Pass Rate: 100% (3/3)
+🔀 Parallel Efficiency: 2.1 avg threads (max 3)
+📈 Trust Level: High (2 auto-approvals earned)
+
+Improvement vs. last feature: +32% tool calls, -15% duration
+```
+
+#### 2. Context Analytics
+
+```bash
+morph-spec analytics context user-auth
+
+Output:
+📊 Context Analytics: user-authentication
+
+Current: 18,200 tokens
+Peak: 23,450 tokens (checkpoint 2)
+Average: 15,800 tokens
+
+Optimization Opportunities:
+  ✅ MCP hygiene: -8K tokens (remove unused context7)
+  ✅ Switch to micro-agent: -6K (jwt-specialist vs dotnet-senior)
+  ✅ Context bundle: -12K (replay from checkpoint 1)
+
+Potential: 18,200 → 5,200 tokens (-71%)
+```
+
+#### 3. Project-Wide Analytics
+
+```bash
+morph-spec analytics project --period=30d
+
+Output:
+📊 MORPH-SPEC Analytics Dashboard
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+🎯 PROJECT: MyApp (Last 30 days)
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+📈 Throughput
+   Total Features:         12
+   Completed:              9 (75%)
+   Avg Duration:           4.2 hours
+   Total Tool Calls:       3,247
+
+🔧 Tool Calls Trend
+   Week 1:  542 calls  ████████░░
+   Week 2:  687 calls  ███████████░
+   Week 3:  891 calls  ████████████████
+   Week 4:  1127 calls ████████████████████  ← +26% vs Week 3
+
+✅ Checkpoint Pass Rate
+   Overall:     92%  ████████████████████░░░░
+   Architecture: 95%  █████████████████████░░░
+   Security:    89%  ██████████████████░░░░░░
+   Design Sys:  91%  ███████████████████░░░░
+
+🔀 Parallelization
+   Avg Concurrent Threads:  2.3
+   Max Concurrent:          5 (user-auth feature)
+   Parallel Efficiency:     78%
+
+📊 Top Performers
+   1. dotnet-senior      847 tool calls, 96% pass rate
+   2. blazor-builder     623 tool calls, 94% pass rate
+   3. ef-modeler         412 tool calls, 98% pass rate
+
+🎯 Trust Progression
+   Features at "low" trust:     3 (25%)
+   Features at "medium" trust:  5 (42%)
+   Features at "high" trust:    4 (33%)
+   Auto-approvals saved:        18 hours
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+💡 Recommendations
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+→ Increase parallelization to 3-4 threads (current: 2.3)
+→ Promote user-auth to "high" trust (95% pass rate)
+→ Review security validator (89%, below 90% threshold)
+```
+
+#### 4. Artifact Trail System
+
+**Source:** 4-layer-agentic-stack-analysis.md (screenshot trails for debugging)
+
+```
+.morph/features/user-auth/artifacts/
+├── checkpoint-1/
+│   ├── screenshots/
+│   │   ├── spec-review-2026-02-18-10-15.png
+│   │   └── architecture-diagram.png
+│   ├── intermediate/
+│   │   ├── spec-draft-v1.md
+│   │   ├── spec-draft-v2.md
+│   │   └── contracts-draft.cs
+│   └── reasoning/
+│       ├── agent-reasoning-dotnet-senior.log
+│       └── validator-reasoning-architecture.log
+│
+├── checkpoint-2/
+│   ├── failures/
+│   │   ├── validation-failure-1.log
+│   │   ├── validation-failure-2.log
+│   │   └── retry-reasoning.log
+│   └── screenshots/
+│       └── validator-output.png
+│
+└── metadata.json
+```
+
+#### 5. Core Four Tracking
+
+```bash
+morph-spec core-four user-auth
+
+Output:
+📊 Core Four: user-authentication
+
+Context: 12,450 tokens (23% of max 54K)
+  ├─ CLAUDE.md: 500 tokens (primed)
+  ├─ Standards: 3,200 tokens (3 files)
+  ├─ Templates: 800 tokens (5 templates)
+  ├─ Spec: 2,100 tokens
+  ├─ Tasks: 1,400 tokens
+  └─ Conversation: 4,450 tokens
+
+Model: Sonnet 4.5 (128K context window)
+
+Prompt: "Implement user authentication with JWT tokens and BCrypt password hashing"
+
+Tools (48 total calls):
+  ├─ Read: 23 (48%)
+  ├─ Write: 8 (17%)
+  ├─ Edit: 12 (25%)
+  └─ Bash: 5 (10%)
+```
+
+### Impact
+
+- **Visibility:** From 0% to 100% (tool calls, duration, efficiency)
+- **Data-driven decisions:** Which agents to use based on pass rates
+- **Debugging:** Artifact trails show exactly what happened at each checkpoint
+- **Optimization:** Context analytics suggest 30-40% token savings
+- **Trust:** Track record enables auto-approvals (50% overhead reduction)
+
+---
+
+## Workflows Evolution
+
+**Source:** All 6 documents (parallel, fusion, long-running, zero-touch patterns)
+
+### Current Workflows (5 types) ✅
+
+| Workflow | Phases | Status |
+|----------|--------|--------|
+| fast-track | Quick Analysis → Implement | ✅ Working |
+| standard | Proposal → Design → Tasks → Implement | ✅ Working |
+| full-morph | All 9 phases | ✅ Working |
+| design-impl | Prototype Analysis → Tasks → Implement | ✅ Working |
+| ui-refresh | Visual Analysis → Design System → Implement | ✅ Working |
+
+### New Workflows (3 types) ❌→✅
+
+#### 1. Fusion Workflow (NEW)
+
+**Purpose:** Rapid prototyping with confidence (best-of-3 designs)
+
+**Pattern:**
+```
+User Request
+  ↓
+Workflow Detection (detects: high uncertainty, critical decision, rapid prototype needed)
+  ↓
+Fusion Executor spawns 3 agents in parallel
+  ├─ Agent 1 (Opus 4.6): Design approach A
+  ├─ Agent 2 (Sonnet 4.5): Design approach B
+  └─ Agent 3 (Sonnet 4.5): Design approach C
+  ↓
+Aggregator scores results
+  ├─ Checkpoint pass rate (40% weight)
+  ├─ Code quality (30% weight)
+  └─ Completeness (30% weight)
+  ↓
+Present best result to user (or all 3 for manual selection)
+  ↓
+Continue with standard/full-morph workflow using winning design
+```
+
+**Impact:**
+- 3x chance one design is optimal
+- Parallel execution (3 agents simultaneously)
+- Higher confidence in critical decisions
+
+#### 2. Long-Running Workflow (NEW)
+
+**Purpose:** Autonomous multi-hour runs with stop hooks
+
+**Pattern:**
+```
+User Request
+  ↓
+Workflow Detection (detects: large scope, >10 files, "autonomous" keyword)
+  ↓
+L-Thread Executor with stop hooks
+  ├─ Agent works autonomously (2-4 hours)
+  ├─ Every 30 min: Stop hook validates progress
+  │   ├─ If validation passes: Continue
+  │   └─ If validation fails: Provide feedback → Continue with feedback
+  ├─ Max retries: 5
+  └─ Final validation before completion
+  ↓
+User reviews only final result (not every checkpoint)
+```
+
+**Impact:**
+- Hours-long autonomous runs (vs 30 min current)
+- Agent self-validates (closed loop)
+- User intervenes only on final review
+
+#### 3. Zero-Touch Workflow (NEW)
+
+**Purpose:** Fully autonomous execution with no approval gates (endgame)
+
+**Requirements:**
+- Trust level: "maximum" (95%+ checkpoint pass rate)
+- Feature type: Routine work (similar to past features)
+- Validation: All checkpoints must pass
+- Rollback: Git revert available if issues
+
+**Pattern:**
+```
+User Request
+  ↓
+Workflow Detection (detects: high trust level, routine feature)
+  ↓
+Auto-approve design gate (based on trust history)
+  ↓
+Auto-approve tasks gate (based on trust history)
+  ↓
+Implement with auto-checkpoints
+  ↓
+Auto-commit if all validations pass
+  ↓
+Notify user: "Feature complete, review at your convenience"
+```
+
+**Safety:**
+- Only enabled if 95%+ past checkpoint pass rate
+- Full validation pipeline still runs
+- Rollback available via `morph-spec rollback user-pagination`
+
+**Impact:**
+- 0% approval overhead for routine features
+- Path to "vibe coding" at industrial scale
+- User reviews asynchronously (not blocking)
+
+### Impact
+
+- **Workflows:** 5 → 8 types (+60%)
+- **Confidence:** Best-of-3 fusion for critical decisions
+- **Autonomy:** Hours-long runs with stop hooks
+- **Efficiency:** Zero-touch for routine features (0% overhead)
+
+---
+
+## Summary
+
+MORPH-SPEC v3.0 features transform the platform across 5 dimensions:
+
+1. **Execution:** 7 thread types (Base → Z-Thread) for complete flexibility
+2. **Context:** R&D framework with 12 techniques for 30-40% token reduction
+3. **Meta-Programming:** 12+ HOPs for reusable, validated prompt patterns
+4. **Analytics:** Comprehensive dashboards (thread, context, project-wide, Core Four)
+5. **Workflows:** 8 workflows (5 current + 3 new) for all use cases
+
+Next: See [AGENTS.md](./AGENTS.md) for agent ecosystem expansion.
+
+---
+
+*MORPH-SPEC v3.0 Features - by Polymorphism Tech*