@polymorphism-tech/morph-spec 3.2.0 → 4.2.0

package/docs/v3.0/ARCHITECTURE.md

@@ -0,0 +1,436 @@
+# MORPH-SPEC v3.0 - Architecture
+
+> Architectural vision, domain reorganization, and complete integration map
+
+**Status:** Design Complete ✅
+**Last Updated:** 2026-02-18
+
+---
+
+## Table of Contents
+
+- [Architectural Vision](#architectural-vision)
+- [Domain Reorganization](#domain-reorganization)
+- [Design Principles](#design-principles)
+- [Integration Map](#integration-map)
+- [Dependency Matrix](#dependency-matrix)
+
+---
+
+## Architectural Vision
+
+### North Star
+
+Transform MORPH-SPEC from a sequential, single-threaded process into a **parallel, context-optimized, self-improving agentic platform** with 10-15x throughput, 30-40% token reduction, and 50% automation.
+
+### Three Pillars
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│           MORPH-SPEC v3.0 - Three Pillars                   │
+└─────────────────────────────────────────────────────────────┘
+
+1️⃣ SCALE Through Parallelization
+   ├─ P-Thread: 3-5 agents working simultaneously
+   ├─ F-Thread: Best-of-N fusion for rapid prototyping
+   └─ Thread Manager: Orchestrate 10-15 parallel threads
+
+2️⃣ OPTIMIZE Through Context Engineering
+   ├─ Reduce: MCP hygiene, context priming (23K → 500 tokens)
+   ├─ Delegate: Sub-agents, background agents, micro-agents
+   └─ Analytics: Real-time token tracking, optimization suggestions
+
+3️⃣ IMPROVE Through Meta-Programming
+   ├─ Higher-Order Prompts (HOPs): Reusable prompt patterns
+   ├─ Thread Analytics: Data-driven agent selection
+   └─ Trust Levels: Auto-approve based on track record (Z-Thread)
+```
+
+#### Pillar 1: SCALE Through Parallelization
+
+**Goal:** Match Boris Churnney's workflow (10-15 parallel threads)
+
+**Key Components:**
+- Thread management system tracking concurrent execution
+- P-Thread executor for parallel agent spawning
+- F-Thread executor for fusion patterns (best-of-3)
+- Thread coordinator managing dependencies
+
+**Impact:**
+- Throughput: 1 → 3-5 features/day (+200-400%)
+- Tool calls: ~100 → ~250 per feature (+150%)
+
+#### Pillar 2: OPTIMIZE Through Context Engineering
+
+**Goal:** Reduce token usage by 30-40% via R&D Framework
+
+**Key Components:**
+- Context priming (CLAUDE.md: 23K → 500 tokens)
+- Context bundles (session replay: 180K → 15K tokens)
+- MCP hygiene (remove unused servers: -10K tokens)
+- Micro-agents (ultra-specialized, 500 tokens context)
+
+**Impact:**
+- CLAUDE.md: -95% tokens
+- Session replay: -92% tokens
+- Response speed: 2-3x faster
+
+#### Pillar 3: IMPROVE Through Meta-Programming
+
+**Goal:** Enable data-driven decisions and automation
+
+**Key Components:**
+- 12+ Higher-Order Prompts (reusable patterns)
+- Thread analytics dashboard (tool calls, duration, efficiency)
+- Context analytics (real-time optimization suggestions)
+- Trust levels (auto-approve based on track record)
+
+**Impact:**
+- Approval overhead: 100% manual → 50% auto
+- Consistency: 100% agent spawns use validated templates
+
+---
+
+## Domain Reorganization
+
+### Current Problem ❌
+
+Domains nested under stacks (blazor-azure, nextjs-supabase):
+- DevX (CLI, workflows, templates) buried in details
+- Cross-cutting concerns (security, observability) fragmented
+- AI/agents treated as "backend" child
+
+### Solution: Tier 0 DevX + 9 Logical Domains ✅
+
+```
+🎯 Tier 0: DEVELOPER EXPERIENCE (Elevated - core of MORPH)
+   ├─ CLI Tools (morph-spec commands)
+   ├─ Workflows (fast-track, standard, full-morph, design-impl, ui-refresh)
+   ├─ Templates (66 Handlebars templates)
+   └─ Standards & Validation (48 standards, 5 validators)
+
+📱 Tier 1: APPLICATION LAYER
+   ├─ Frontend (Next.js, Blazor)
+   └─ Backend (.NET 10)
+
+🤖 Tier 1: AI & AGENT ORCHESTRATION (Elevated domain)
+   ├─ Microsoft Agent Framework
+   ├─ RAG Pipeline
+   └─ Prompt Engineering
+
+🗄️ Tier 1: DATA & PERSISTENCE (Unified)
+   ├─ Relational (EF Core, Azure SQL)
+   ├─ Vector/Search (Azure AI Search)
+   └─ NoSQL/Cache (Cosmos, Redis, Blob)
+
+🔐 Tier 1: SECURITY & IDENTITY (Cross-cutting)
+   ├─ Authentication (Passkeys, Clerk, Azure Identity)
+   ├─ Authorization (RBAC, Policy-based)
+   └─ Secrets Management (Key Vault)
+
+🔄 Tier 1: INTEGRATION & COMMUNICATION
+   ├─ APIs (REST, GraphQL, gRPC)
+   ├─ Event-Driven (Service Bus, CQRS, Event Sourcing)
+   ├─ Background Processing (Hangfire)
+   └─ External Services (Asaas, Resend)
+
+☁️ Tier 1: INFRASTRUCTURE & PLATFORM
+   ├─ Azure Cloud (Container Apps, SQL, Storage)
+   ├─ IaC (Bicep)
+   ├─ Containers (Docker)
+   └─ CI/CD (GitHub Actions)
+
+📊 Tier 1: OBSERVABILITY (NEW - 0% → 100%)
+   ├─ Monitoring (App Insights, Azure Monitor)
+   ├─ Logging (Serilog, Log Analytics)
+   ├─ Tracing (OpenTelemetry)
+   └─ Metrics (Custom metrics, Alerts)
+
+🧪 Tier 1: QUALITY ASSURANCE
+   ├─ Testing (xUnit, Moq, TestContainers, bUnit)
+   └─ Code Analysis (SOLID, Clean Code)
+
+🏗️ Tier 1: ARCHITECTURE & PATTERNS
+   ├─ Clean Architecture, Vertical Slice
+   ├─ CQRS, Repository, DDD
+   └─ SOLID Principles
+```
+
+### Changes from v2.x
+
+| Change | Before | After | Impact |
+|--------|--------|-------|--------|
+| **DevX Position** | Buried in stack details | Tier 0 (top-level) | CLI/workflows/templates highlighted |
+| **Observability** | Scattered (MAF, Azure) | Own domain (cross-cutting) | 0% → 100% coverage |
+| **AI/Agents** | Child of Backend | Own domain (Tier 1) | Elevated importance |
+| **Data** | Fragmented (EF, SQL, Vector) | Unified domain | Cohesive persistence layer |
+| **Security** | Mixed with Integrations | Cross-cutting domain | Proper separation |
+| **Event-Driven** | Missing | Added to Integration | Event architecture enabled |
+
+### Impact
+
+- **16 agents → 26 agents** (+62%)
+- **0% observability → 100%** (new observability-expert)
+- **DevX elevated** to Tier 0
+- **Cross-cutting concerns** properly separated
+
+---
+
+## Design Principles
+
+### 1. Integration Over Isolation
+
+**Principle:** Every new component must connect to 3+ existing systems.
+
+**Example:**
+- Thread Manager connects to: state.json, agents.json, workflows/*.json, analytics-engine
+- Context Optimizer connects to: standards/*.md, templates/REGISTRY.json, context-tracker, micro-agents
+- Fusion Aggregator connects to: validators, analytics-engine, thread-manager, approval-gates
+
+**Validation:** See [Dependency Matrix](#dependency-matrix) - every component lists 3+ connections.
+
+### 2. Common Themes First
+
+**Principle:** If 3+ framework-comparison documents mention a feature → HIGH priority.
+
+**High Priority (3+ mentions):**
+- ✅ Parallelization (thread-based, 4-layer, context-engineering)
+- ✅ Analytics (thread-based, context-engineering, 4-layer, agents-gaps)
+- ✅ Context Optimization (context-engineering, 4-layer, thread-based)
+- ✅ Meta-Programming (4-layer, thread-based, context-engineering)
+- ✅ Agent Expansion (agents-gaps, framework-hierarchy, thread-based)
+
+**Medium Priority (2 mentions):**
+- Trust & Autonomy (thread-based, context-engineering)
+
+### 3. Backward Compatibility
+
+**Principle:** Existing v2.x workflows/agents/templates continue to work.
+
+**Implementation:**
+- v2.x workflows (fast-track, standard, full-morph, design-impl, ui-refresh) → unchanged
+- v3.0 adds 3 new workflows (fusion, long-running, zero-touch) → optional
+- Migration tool: `morph-spec migrate v2-to-v3` (schema updates only)
+- Rollback available: `morph-spec migrate rollback-to-v2`
+
+### 4. Measurable Impact
+
+**Principle:** Every feature has quantifiable success metrics.
+
+**Examples:**
+- Thread Management: Tool calls per feature, duration, parallel efficiency
+- Context Optimization: Token reduction %, response speed improvement
+- Trust Levels: Auto-approval rate, checkpoint pass rate
+- Analytics: Throughput (features/day), improvement trends
+
+### 5. No Loose Ends
+
+**Principle:** Complete dependency graph, all systems interconnected.
+
+**Validation:**
+- 60+ features from 6 documents → All addressed ✅
+- 15-step integration flow → Every step connects to next ✅
+- Dependency matrix → Every component lists connections ✅
+
+---
+
+## Integration Map
+
+### Complete 16-Step Flow
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│           MORPH-SPEC v3.0 - Complete Integration Map         │
+└──────────────────────────────────────────────────────────────┘
+
+USER REQUEST
+    ↓
+[1] WORKFLOW DETECTOR
+    ├─ Analyzes: keywords, complexity, trust level
+    ├─ Connects to: workflows/*.json configs, state.json (trust history)
+    └─ Outputs: workflow type (fast-track/standard/full-morph/fusion/long-running/zero-touch)
+    ↓
+[2] THREAD MANAGER
+    ├─ Creates thread(s) based on workflow
+    ├─ Connects to: state.json (thread tracking), agents.json (agent selection)
+    └─ Outputs: thread IDs, parallel coordination plan
+    ↓
+[3] CONTEXT OPTIMIZER
+    ├─ Primes context based on feature type
+    ├─ Connects to: standards/*.md (selective loading), templates/REGISTRY.json
+    └─ Outputs: Primed context (500 tokens vs 23K)
+    ↓
+[4] AGENT EXECUTOR
+    ├─ Spawns agents using HOPs
+    ├─ Connects to: meta-prompts/*.md (squad leaders, workers), agents.json (hierarchy)
+    └─ Outputs: Agent tasks
+    ↓
+[5] PARALLEL EXECUTOR (if P-Thread)
+    ├─ Runs 3-5 agents simultaneously
+    ├─ Connects to: thread-manager (coordination), state.json (status tracking)
+    └─ Outputs: Parallel results
+    ↓
+[6] FUSION AGGREGATOR (if F-Thread)
+    ├─ Scores N results, picks best
+    ├─ Connects to: validators (checkpoint scoring), analytics (quality metrics)
+    └─ Outputs: Winning design
+    ↓
+[7] VALIDATORS (Tier 4)
+    ├─ Runs architecture, security, packages, design-system validators
+    ├─ Connects to: hooks/agent-teams/*.js (auto-triggered), state.json (checkpoint history)
+    └─ Outputs: Pass/fail + violations
+    ↓
+[8] CHECKPOINTS
+    ├─ Every 3 tasks, runs full validation
+    ├─ Connects to: validators, artifact-trail (saves screenshots/logs), stop-hooks (if L-Thread)
+    └─ Outputs: Checkpoint pass/fail
+    ↓
+[9] STOP HOOKS (if L-Thread)
+    ├─ Agent attempts to stop → validation runs
+    ├─ Connects to: validators, checkpoints, thread-manager (continue/complete)
+    └─ Outputs: Continue (with feedback) or Complete
+    ↓
+[10] TRUST MANAGER
+    ├─ Tracks checkpoint pass rates
+    ├─ Connects to: state.json (trust history), approval gates (auto-approve if high trust)
+    └─ Outputs: Trust level (low/medium/high/maximum)
+    ↓
+[11] APPROVAL GATES
+    ├─ Design gate, tasks gate
+    ├─ Connects to: trust-manager (auto-approve?), state.json (gate status)
+    └─ Outputs: Approved/rejected/auto-approved
+    ↓
+[12] ANALYTICS ENGINE
+    ├─ Tracks tool calls, duration, parallel efficiency
+    ├─ Connects to: thread-manager, context-tracker, validators (pass rates)
+    └─ Outputs: Dashboards, recommendations
+    ↓
+[13] CONTEXT TRACKER
+    ├─ Monitors token usage real-time
+    ├─ Connects to: context-optimizer (suggestions), analytics-engine (metrics)
+    └─ Outputs: Context analytics, optimization opportunities
+    ↓
+[14] ARTIFACT TRAIL
+    ├─ Saves screenshots, intermediate files, agent reasoning
+    ├─ Connects to: checkpoints (triggered on fail), state.json (artifact paths)
+    └─ Outputs: Debugging trail
+    ↓
+[15] LEARNING SYSTEM
+    ├─ Captures patterns (success/avoid/optimization)
+    ├─ Connects to: state.json (feature outcomes), standards/*.md (promoted patterns)
+    └─ Outputs: patterns-learned.md entries
+    ↓
+[16] GIT HOOKS
+    ├─ Pre-commit, commit-msg, pre-push
+    ├─ Connects to: validators, orchestrator.sh, test suite
+    └─ Outputs: Commit allowed/blocked
+    ↓
+OUTPUT: Feature Deliverables + Analytics + Artifacts
+```
+
+---
+
+## Dependency Matrix
+
+Every new v3.0 component connects to 3+ existing systems:
+
+| Component | Connects To (3+) | Purpose |
+|-----------|------------------|---------|
+| **Thread Manager** | state.json, agents.json, workflows/*.json, analytics-engine | Track concurrent execution, select agents |
+| **Context Optimizer** | standards/*.md, templates/REGISTRY.json, context-tracker, micro-agents | Selective context loading, token reduction |
+| **Parallel Executor** | thread-manager, state.json, agents.json, fusion-aggregator | Concurrent agent spawning, coordination |
+| **Fusion Aggregator** | validators, analytics-engine, thread-manager, approval-gates | Score N designs, pick best |
+| **Stop Hooks** | validators, checkpoints, thread-manager, artifact-trail | Autonomous validation loops |
+| **Trust Manager** | state.json, checkpoints, approval-gates, analytics-engine | Track record, auto-approval |
+| **Analytics Engine** | thread-manager, context-tracker, validators, state.json | Metrics collection, dashboards |
+| **Context Tracker** | context-optimizer, analytics-engine, state.json, MCP-servers | Token usage monitoring |
+| **Artifact Trail** | checkpoints, state.json, validators, thread-manager | Debugging screenshots, logs |
+| **HOPs** | agents.json, templates/REGISTRY.json, standards/*.md, parallel-executor | Reusable prompt templates |
+
+**Validation:** ✅ All components connect to 3+ systems → No isolation.
+
+---
+
+## State Schema Extensions
+
+### state.json v3.0 Schema
+
+```json
+{
+  "$schema": "./templates/state.template.json",
+  "version": "3.0.0",
+
+  "threads": {
+    "{thread-id}": {
+      "id": "string",
+      "feature": "string",
+      "agent": "string",
+      "type": "base|parallel|fusion|long-running|zero-touch",
+      "status": "pending|running|completed|failed",
+      "startedAt": "ISO8601",
+      "completedAt": "ISO8601?",
+      "metrics": {
+        "toolCalls": "number",
+        "duration": "string",
+        "estimatedCompletion": "ISO8601?"
+      }
+    }
+  },
+
+  "features": {
+    "{feature-name}": {
+      "...existing fields...",
+
+      "threadMetrics": {
+        "totalToolCalls": "number",
+        "threadDuration": "string",
+        "maxParallelThreads": "number",
+        "avgParallelThreads": "number",
+        "checkpointPassRate": "number (0-1)"
+      },
+
+      "trustConfig": {
+        "level": "low|medium|high|maximum",
+        "autoApprove": {
+          "design": "boolean",
+          "tasks": "boolean"
+        },
+        "reason": "string",
+        "history": [
+          {
+            "feature": "string",
+            "checkpointPassRate": "number",
+            "timestamp": "ISO8601"
+          }
+        ]
+      },
+
+      "contextBundles": [
+        {
+          "bundleId": "string",
+          "checkpoint": "number",
+          "timestamp": "ISO8601",
+          "path": "string"
+        }
+      ]
+    }
+  }
+}
+```
+
+---
+
+## Summary
+
+MORPH-SPEC v3.0 architecture is designed around three core pillars (Scale, Optimize, Improve) with:
+
+- **9 logical domains** (DevX to Tier 0, 100% coverage)
+- **Complete integration** (16-step flow, every component connects to 3+)
+- **5 design principles** (Integration, Common Themes, Backward Compat, Measurable, No Loose Ends)
+- **Zero loose ends** (all 60+ features from 6 docs addressed)
+
+Next: See [FEATURES.md](./FEATURES.md) for detailed feature descriptions.
+
+---
+
+*MORPH-SPEC v3.0 Architecture - by Polymorphism Tech*