agentic-orchestrator 0.1.26 → 0.1.28

package/spec-files/outstanding/execution_mode_critical_review.md

@@ -0,0 +1,355 @@
+# Critical Architectural Review: Execution Mode Specification
+
+**Reviewer:** Senior Software Architect (Critical Perspective)  
+**Date:** 2026-03-05  
+**Spec:** `agentic_orchestrator_execution_mode_spec.md`  
+**Verdict:** ⚠️ **MAJOR REVISIONS REQUIRED** - Original design had critical flaws
+
+---
+
+## 🔥 FATAL FLAWS IDENTIFIED
+
+### 1. Race Conditions Everywhere
+
+**Problem:** Watchdog monitors while agent writes. No coordination.
+
+**Original Design:** Agent writes freely, checkpoint runs whenever triggered.
+
+**Critical Issues:**
+
+- Checkpoint validation runs while agent is mid-write
+- Multiple checkpoints could trigger simultaneously
+- File system events arrive out of order
+- No write transaction boundaries
+
+**Fix Applied:** Agent pause/resume protocol with acknowledgment, checkpoint serialization, debouncing.
+
+---
+
+### 2. Validation is Too Late
+
+**Problem:** Interactive mode validates AFTER changes are written.
+
+**Original Design:** Agent writes → Checkpoint captures diff → Validate → Maybe revert.
+
+**Critical Issues:**
+
+- Files already modified (potential corruption)
+- Agent made dependent changes based on invalid state
+- Revert is destructive and loses work
+- No atomic rollback guarantee
+
+**Fix Applied:** Shadow workspace strategy - agent writes to isolated copy, validation promotes to real worktree only if valid.
+
+---
+
+### 3. No Rollback Strategy
+
+**Problem:** "Optional revert with `git checkout`" is naive.
+
+**Original Design:** If validation fails, run `git checkout -- <files>`.
+
+**Critical Issues:**
+
+- What if agent has uncommitted changes across 50 files?
+- What if some changes valid, some invalid?
+- What if agent still writing when revert happens?
+- What about file system state (temp files, build artifacts)?
+
+**Fix Applied:** Three rollback strategies:
+
+1. Full checkpoint rollback (restore to exact checkpoint state)
+2. Partial file rollback (revert only violated files)
+3. Smart rollback (keep valid changes, revert only violations)
+
+---
+
+### 4. Checkpoint Overhead is Unbounded
+
+**Problem:** Every checkpoint runs full `git diff` + validation.
+
+**Original Design:** No optimization, no caching, no timeouts.
+
+**Critical Issues:**
+
+- With 1000 files: parsing takes seconds
+- Complex plan patterns: validation is slow
+- No timeout budget
+- Blocks agent progress
+
+**Fix Applied:**
+
+- Validation cache (per file + plan version)
+- Incremental diff (only changed files since last checkpoint)
+- Timeout budgets (5s max, then degrade)
+- Sampling for large changesets (>100 files)
+
+---
+
+### 5. Concurrent Feature Isolation is Broken
+
+**Problem:** "Each feature has isolated worktree" but services are singletons.
+
+**Original Design:** Shared watchdog service, no concurrency control.
+
+**Critical Issues:**
+
+- Watchdog service has shared state
+- Checkpoint service has no concurrency control
+- Validation service could be bottlenecked
+- File system events could be misattributed
+
+**Fix Applied:**
+
+- Per-feature watchdog instances (separate `chokidar` per feature)
+- Checkpoint serialization per feature
+- Validation queue with priority and concurrency limits
+- Backpressure (max 5 concurrent interactive features)
+
+---
+
+### 6. Agent Notification is Fire-and-Forget
+
+**Problem:** "Send message via WorkerProvider.sendMessage" assumes agent listens.
+
+**Original Design:** No acknowledgment, no retry, no escalation.
+
+**Critical Issues:**
+
+- Agent may not be listening
+- Agent may not respond
+- Agent may not understand message format
+- Agent could ignore critical violations
+
+**Fix Applied:** Structured message protocol with:
+
+- Acknowledgment requirement for critical messages
+- Timeout and retry logic
+- Escalation to human on repeated failures
+- Agent pause on critical violations
+
+---
+
+### 7. No Graceful Degradation
+
+**Problem:** If watchdog/checkpoint/validation fails, what happens?
+
+**Original Design:** No error handling, no fallback, no recovery.
+
+**Critical Issues:**
+
+- Agent continues running blind
+- Changes accumulate without validation
+- System state becomes inconsistent
+- No recovery path
+
+**Fix Applied:**
+
+- Circuit breakers (3 failures → escalate to human)
+- Health checks for all services
+- Automatic fallback to deterministic mode
+- Agent timeout with forced checkpoint
+- Recovery protocol with state capture
+
+---
+
+### 8. Audit Trail is Incomplete
+
+**Problem:** Checkpoints only capture state at intervals.
+
+**Original Design:** Only checkpoint snapshots, no intermediate events.
+
+**Critical Issues:**
+
+- No record of intermediate states
+- No record of attempted-but-reverted files
+- No record of agent's decision process
+- Cannot reconstruct exact sequence
+
+**Fix Applied:**
+
+- Continuous event log (every file change with timestamp)
+- Checkpoint chains (linked list of checkpoints)
+- Incremental + cumulative diffs
+- Validation history log
+
+---
+
+### 9. Security Nightmare
+
+**Problem:** Agent has direct file system access with only periodic validation.
+
+**Original Design:** No sandboxing, no resource limits, no path validation.
+
+**Critical Issues:**
+
+- Agent could write malicious code that executes before checkpoint
+- Agent could modify `.git` directory
+- Agent could escape worktree via symlinks
+- Agent could exhaust disk space
+
+**Fix Applied:**
+
+- Path canonicalization and validation
+- `.git` directory protection (immediate block)
+- Symlink detection and removal
+- Disk quota per feature (1GB limit)
+- Executable file detection and flagging
+
+---
+
+### 10. Performance Assumptions are Unvalidated
+
+**Problem:** "Checkpoint validation completes in < 500ms" - based on what?
+
+**Original Design:** No benchmarks, no budgets, no degradation strategy.
+
+**Critical Issues:**
+
+- What if diff is 10,000 lines?
+- What if plan has 100 allowed_areas patterns?
+- What if lock service is slow?
+- What if file system is network-mounted?
+
+**Fix Applied:**
+
+- Explicit latency budgets with targets and maximums
+- Timeout-based degradation strategies
+- Throughput requirements
+- Resource limits per feature and globally
+- Monitoring metrics and alert thresholds
+
+---
+
+## ✅ IMPROVEMENTS APPLIED
+
+### Architecture Enhancements
+
+1. **Shadow Workspace Strategy**
+   - Agent writes to isolated shadow directory
+   - Validation before promotion to real worktree
+   - Atomic promotion or discard
+   - Configurable per feature or globally
+
+2. **Agent Communication Protocol**
+   - Structured message format (JSON)
+   - Pause/resume with acknowledgment
+   - Violation notifications with severity
+   - Timeout and retry logic
+
+3. **Rollback Capabilities**
+   - Full checkpoint rollback
+   - Partial file rollback
+   - Smart rollback (keep valid, revert violations)
+   - Inverse diff application
+
+4. **Performance Budgets**
+   - Latency targets: checkpoint < 500ms, validation < 1s
+   - Throughput targets: 100 file changes/sec
+   - Resource limits: 1GB disk per feature, 50MB memory per watchdog
+   - Degradation strategies for all timeouts
+
+5. **Concurrency Control**
+   - Per-feature service instances
+   - Validation queue with priority
+   - Backpressure (max 5 concurrent interactive features)
+   - Event attribution per feature
+
+6. **Security Hardening**
+   - Path validation and canonicalization
+   - `.git` directory protection
+   - Symlink detection and blocking
+   - Disk quota enforcement
+   - Executable file detection
+
+7. **Graceful Degradation**
+   - Circuit breakers (3 failures → escalate)
+   - Health checks for all services
+   - Automatic fallback to deterministic mode
+   - Recovery protocol with state capture
+
+8. **Audit Trail Completeness**
+   - Continuous event log (JSONL format)
+   - Checkpoint chains (linked list)
+   - Incremental + cumulative diffs
+   - Validation history
+
+9. **Monitoring and Observability**
+   - Checkpoint latency (p50, p95, p99)
+   - Validation latency and cache hit rate
+   - Checkpoint failure rate
+   - Disk/memory usage per feature
+   - Alert thresholds
+
+10. **Provider Capability Detection**
+    - Query provider for interactive mode support
+    - Automatic fallback if unsupported
+    - Clear error messages
+    - Standardized capability interface
+
+---
+
+## 📊 SPEC METRICS
+
+| Metric                   | Before | After      | Change |
+| ------------------------ | ------ | ---------- | ------ |
+| Total lines              | 951    | 1805       | +90%   |
+| Risk sections            | 4      | 9          | +125%  |
+| Mitigation strategies    | 8      | 27         | +238%  |
+| Code examples            | 12     | 28         | +133%  |
+| Performance requirements | 0      | 4 sections | NEW    |
+| Security mitigations     | 1      | 5          | +400%  |
+
+---
+
+## 🎯 RECOMMENDATION
+
+**Status:** ✅ **APPROVED WITH REVISIONS**
+
+The original design was fundamentally flawed for production use. The revised spec addresses all critical issues and is now ready for implementation with the following caveats:
+
+1. **Start with Strategy A (Direct Worktree)** for simplicity, but implement Strategy B (Shadow Workspace) infrastructure from day one for easy migration.
+
+2. **Implement circuit breakers and health checks first** before any interactive mode execution.
+
+3. **Performance budgets are mandatory** - do not ship without timeout and degradation strategies.
+
+4. **Security hardening is non-negotiable** - path validation, `.git` protection, and disk quotas must be in place before beta.
+
+5. **Monitor everything** - checkpoint latency, validation failures, disk usage, memory usage. Set up alerts before production.
+
+---
+
+## 📋 IMPLEMENTATION PRIORITY
+
+### P0 (Must Have Before Any Interactive Execution)
+
+- Agent pause/resume protocol
+- Path validation and security hardening
+- Circuit breakers and health checks
+- Checkpoint serialization (no concurrent checkpoints per feature)
+
+### P1 (Must Have Before Beta)
+
+- Shadow workspace strategy implementation
+- Rollback strategies (full, partial, smart)
+- Performance budgets and timeouts
+- Monitoring and alerts
+
+### P2 (Must Have Before Production)
+
+- Validation cache
+- Continuous event log
+- Provider capability detection
+- Graceful degradation and fallback
+
+### P3 (Nice to Have)
+
+- Validation sampling for large changesets
+- Real-time dashboard streaming
+- Checkpoint comparison UI
+- Advanced rollback commands
+
+---
+
+**Conclusion:** The spec is now production-ready with comprehensive risk mitigation. Original design would have failed catastrophically in production. Revised design is robust, safe, and performant.