dhurandhar 1.0.0

package/docs/SYSTEM_OBSERVER.md

@@ -0,0 +1,433 @@
+# System Observer & Architectural Drift Detection
+
+## Overview
+
+Dhurandhar v2.3 introduces the **System Observer** persona and **Architectural Drift Detection** system to manage the delta between "Intended State" (SDM) and "Current State" (codebase).
+
+## Philosophy
+
+### Traditional Approach (Anti-Pattern)
+
+```
+Developer: "I have 5 services in my codebase"
+Framework: "Great! Which ones?"
+Developer: "auth, user, order, payment, inventory"
+Framework: "Let me add those to the design document..."
+
+[Later...]
+
+Developer: "I added a notification service"
+Framework: "You should update the design document"
+Developer: "I'll do it later..."
+
+[Design document becomes stale, never matches reality]
+```
+
+### Dhurandhar Approach
+
+```
+Developer: "I have 5 services in my codebase"
+
+System Observer:
+[Scans codebase]
+[Reads SYSTEM_DESIGN_MAP.yaml]
+"Found 5 services in code, 3 in SDM"
+
+Unmanaged (in code, not in SDM):
+  - notification-service
+  - analytics-service
+
+Sync SDM to match? (y/n)
+  → y
+
+✓ SDM synchronized
+```
+
+**Direct action. No stale documentation.**
+
+## System Observer Persona
+
+### Role
+
+- **Technical Auditor**: Compare SDM vs codebase
+- **State Synchronizer**: Keep SDM aligned with reality
+- **Session Gatekeeper**: First persona invoked on session start
+- **Evidence-Based Reporter**: Only report verifiable facts
+
+### Responsibilities
+
+1. **Drift Detection**:
+   - Identify unimplemented (SDM → code gaps)
+   - Identify unmanaged (code → SDM gaps)
+   - Detect strategy violations
+
+2. **Session Rehydration**:
+   - Load SDM on session start
+   - Perform background audit
+   - Provide context to other personas
+
+3. **Compliance Reporting**:
+   - Service implementation status
+   - Entity implementation status
+   - Story test coverage
+   - Strategy compliance percentage
+
+## Drift Types
+
+### 1. Unimplemented (SDM Ahead of Code)
+
+**Definition**: Defined in `SYSTEM_DESIGN_MAP.yaml` but missing in codebase
+
+**Examples**:
+- Service defined in SDM but no service directory exists
+- Entity defined but no database schema/model file
+- Story defined but tests not generated
+- API endpoint in SDM but no route handler in code
+
+**Detection**:
+```
+SDM: services: [auth-service, user-service, order-service]
+Code: services/ directory contains [auth-service, user-service]
+
+Unimplemented: order-service
+```
+
+### 2. Unmanaged (Code Ahead of SDM)
+
+**Definition**: Exists in codebase but not in `SYSTEM_DESIGN_MAP.yaml`
+
+**Examples**:
+- Service directory exists but not listed in SDM
+- Database table not mapped to any entity
+- API routes not documented in stories
+- Code implementation without corresponding design
+
+**Detection**:
+```
+Code: services/ directory contains [auth, user, notification]
+SDM: services: [auth-service, user-service]
+
+Unmanaged: notification
+```
+
+### 3. Strategy Violations (Implementation Contradicts Strategies)
+
+**Definition**: Code implementation violates active `technical_strategies`
+
+**Examples**:
+- Shared database when `persistence.model: database_per_service`
+- Synchronous calls when `communication.primary_pattern: asynchronous_events`
+- No circuit breaker when `resilience.circuit_breaker: true`
+- No JWT validation when `security.authentication: jwt_centralized`
+
+**Detection**:
+```
+Strategy: persistence.model = database_per_service
+Service: payment-service
+Code: No dedicated database config found
+
+Violation: payment-service missing dedicated database
+```
+
+## CLI Commands
+
+### 1. Summary Report
+
+```bash
+dhurandhar audit --summary
+```
+
+**Output**:
+```
+📊 System Health Summary
+
+Architectural Alignment:
+  Services: 4/5 implemented
+  Entities: 8/8 implemented
+  Stories: 10/12 tested
+  Strategy Compliance: 90%
+
+Drift Assessment:
+  Drift Percentage: 18%
+  Status: ⚠ Minor drift detected
+
+View details:
+  dhurandhar audit --drift
+```
+
+### 2. Drift Details
+
+```bash
+dhurandhar audit --drift
+```
+
+**Output**:
+```
+🔍 Drift Detection Report
+
+Unimplemented (Defined in SDM, Missing in Code):
+  
+  Services:
+    ✗ order-service: Process customer orders
+    
+  Stories (Tests not generated):
+    ✗ STORY-005: Payment Processing (EPIC-002)
+    ✗ STORY-008: Order Fulfillment (EPIC-003)
+
+Unmanaged (Exists in Code, Not in SDM):
+  
+  Services:
+    ⚠ notification (services/notification/)
+    ⚠ analytics (services/analytics/)
+  
+  Entities:
+    ⚠ Product (models/Product.ts)
+
+Strategy Violations:
+  
+  ⚡ payment-service: Missing event boundaries
+     Strategy: communication
+     Expected: asynchronous_events
+     Actual: no event_boundaries config
+  
+  ⚡ user-service: Missing Redis dependency
+     Strategy: state_management
+     Expected: distributed_redis
+     Actual: no redis dependency
+
+Recommendations:
+  → Codebase is ahead of SDM
+    Run: dhurandhar audit --sync to update SDM
+  
+  → Strategy violations detected
+    Run: dhurandhar strategy --align to fix
+```
+
+### 3. Direct-Action Sync
+
+```bash
+dhurandhar audit --sync
+```
+
+**Interactive**:
+```
+🔄 Sync SDM to Codebase
+
+Sync Analysis:
+  Unimplemented (SDM ahead): 1 services, 0 entities
+  Unmanaged (Code ahead): 2 services, 1 entities
+
+Recommendation: Update SDM to match code
+
+? Sync SDM to codebase state? Yes
+
+✓ SDM synchronized
+
+Changes:
+  + 2 services added to SDM
+  + 1 entities added to SDM
+
+Verify changes:
+  dhurandhar audit --summary
+```
+
+**Direct action. No multi-step interview.**
+
+### 4. Full Audit
+
+```bash
+dhurandhar audit
+```
+
+Combines `--summary` + `--drift` in one report.
+
+## Session Rehydration
+
+### System Observer as Gatekeeper
+
+**Every session starts with System Observer**:
+
+```
+[Session Start]
+
+System Observer:
+1. Load SYSTEM_DESIGN_MAP.yaml
+2. Perform silent background audit
+3. Generate context summary
+4. Calculate drift percentage
+5. Provide context to other personas
+6. Report to user (if drift > 25%)
+```
+
+### Rehydration Report (drift < 25%)
+
+```
+[System Observer] Session Ready
+
+5 services, 8 entities, 12 stories (all tested)
+Strategy Compliance: 95%
+Drift: 8%
+
+Context loaded. What's next?
+```
+
+### Rehydration Report (drift >= 25%)
+
+```
+[System Observer] Session Rehydrated
+
+Project: my-platform
+Architecture: microservices
+
+Active Strategies:
+  Persistence: database_per_service
+  Communication: asynchronous_events (kafka)
+
+Architecture Status:
+  Services: 3/5 implemented
+  Entities: 6/8 implemented
+  Stories: 8/12 tested
+  Strategy Compliance: 80%
+
+Drift: 32%
+
+⚠ Significant architectural drift detected
+
+Unimplemented:
+  - order-service (defined in SDM)
+  - payment-service (defined in SDM)
+  - STORY-005, STORY-008 (tests not generated)
+
+Strategy Violations:
+  - user-service: Missing event boundaries
+
+Run for details:
+  dhurandhar audit --drift
+
+Definition of Done:
+  EPIC-001 (Authentication): 80% (4/5 stories complete)
+  EPIC-002 (Orders): 57% (4/7 stories complete)
+
+What's next?
+```
+
+## Drift Detection Engine
+
+### Implementation
+
+**Location**: `tools/lib/sdm-manager.js`
+
+**Method**: `performAudit()`
+
+**Process**:
+1. Load SDM
+2. Scan codebase:
+   - Service directories (`services/`, `cmd/`, `src/services/`)
+   - Entity files (`models/`, `entities/`, `schema/`)
+   - Test files (`tests/`, `__tests__/`)
+   - Config files (`package.json`, `go.mod`, etc.)
+3. Cross-reference:
+   - SDM services vs codebase services
+   - SDM entities vs codebase entities
+   - SDM stories vs test files
+   - Active strategies vs service configs
+4. Identify gaps:
+   - Unimplemented
+   - Unmanaged
+   - Strategy violations
+5. Calculate drift percentage
+6. Return audit report
+
+### Audit Report Structure
+
+```javascript
+{
+  timestamp: "2024-01-15T10:30:00Z",
+  
+  summary: {
+    total_services: 5,
+    implemented_services: 4,
+    total_entities: 8,
+    implemented_entities: 8,
+    total_stories: 12,
+    tested_stories: 10,
+    strategy_compliance_percentage: 90
+  },
+  
+  unimplemented: {
+    services: [{ name: "order-service", scope: "..." }],
+    entities: [],
+    stories: [{ id: "STORY-005", name: "...", epic: "EPIC-002" }]
+  },
+  
+  unmanaged: {
+    services: [{ name: "notification", path: "services/notification/" }],
+    entities: [{ name: "Product", file: "models/Product.ts" }]
+  },
+  
+  strategy_violations: [
+    {
+      service: "payment-service",
+      strategy: "communication",
+      violation: "Missing event boundaries",
+      expected: "asynchronous_events",
+      actual: "no event_boundaries config"
+    }
+  ],
+  
+  drift_percentage: 18
+}
+```
+
+## Dhurandhar Council Integration
+
+The System Observer is part of the **Dhurandhar Council** - a 4-persona system:
+
+1. **System Observer**: Audit, drift detection, session rehydration
+2. **Lead System Architect**: Service/entity design, strategy management
+3. **Test Architect**: Story translation, test generation
+4. **Edge Case Hunter**: Boundary conditions, security analysis
+
+**Delegation Pattern**:
+```
+System Observer detects drift →
+  Unimplemented service → Lead System Architect
+  Missing tests → Test Architect
+  Strategy violation → Lead System Architect
+```
+
+## Benefits
+
+### 1. Zero Stale Documentation
+
+SDM stays in sync with code through:
+- Automatic drift detection
+- Direct-action sync command
+- Session start audits
+
+### 2. Immediate Context
+
+Every session starts with:
+- Complete architecture state
+- Drift assessment
+- Definition of Done status
+- Strategy compliance
+
+### 3. Evidence-Based Reports
+
+System Observer only reports what it can verify:
+- File existence
+- Directory structure
+- Config file contents
+- No speculation
+
+### 4. Direct Action
+
+`dhurandhar audit --sync` updates SDM without:
+- Multi-step interviews
+- Justification loops
+- Manual editing
+
+---
+
+**Dhurandhar v2.3: Intent (SDM) ↔ Reality (Code) always aligned**