agentic-orchestrator 0.1.25 → 0.1.26

package/spec-files/outstanding/agentic_orchestrator_kernel_simplification_spec.md

@@ -0,0 +1,834 @@
+# Feature Spec: Kernel Simplification and Service Extraction (AOP)
+
+> **Purpose of this document**: Define a systematic refactoring plan to simplify the AopKernel class by extracting cohesive services, reducing its size from 1768 lines to <500 lines, and improving maintainability while preserving all existing behavior and contracts.
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- **Reduce kernel size**: From 1768 lines to <500 lines (70% reduction)
+- **Extract cohesive services**: Move domain logic into focused service classes
+- **Preserve all behavior**: Zero breaking changes to tool contracts, CLI, or runtime
+- **Maintain test coverage**: All existing tests must pass without modification
+- **Improve maintainability**: Each service has single responsibility
+- **Enable parallel development**: Services can be modified independently
+
+### 1.2 Success Criteria
+
+- ✅ Kernel class <500 lines
+- ✅ All 67 tool methods delegated to services
+- ✅ Zero test failures
+- ✅ `npm run verify` passes
+- ✅ All existing tool contracts unchanged
+- ✅ Performance unchanged (±5% latency tolerance)
+
+### 1.3 Non-Goals
+
+- Not changing tool contracts or error codes
+- Not modifying state/index/plan schemas
+- Not replacing file-based storage
+- Not adding new features
+- Not changing CLI commands or flags
+
+---
+
+## 2. Current State Analysis
+
+### 2.1 Kernel Metrics
+
+**File:** `apps/control-plane/src/core/kernel.ts`
+
+- **Total lines:** 1768
+- **Class lines:** 1540 (lines 207-1747)
+- **Methods:** 67 async methods
+- **Dependencies:** 15 service instances
+- **Responsibilities:** 12+ distinct concerns
+
+### 2.2 Current Responsibilities
+
+The kernel currently handles:
+
+1. **Configuration loading** (policy, gates, agents, adapters)
+2. **Schema validation** (all artifact types)
+3. **Tool routing** (67 tool methods)
+4. **Feature lifecycle** (init, discover, context)
+5. **State management** (read, write, patch, locks)
+6. **Index management** (read, write, atomic updates)
+7. **Plan operations** (submit, update, get, collision)
+8. **Patch operations** (validate, apply)
+9. **Gate execution** (run, evidence, selection)
+10. **Lock management** (acquire, release, leases)
+11. **Merge operations** (ready_to_merge, cleanup)
+12. **Reporting** (dashboard, summaries)
+13. **QA index** (get, update)
+14. **Cost tracking** (record, get)
+15. **Run lease** (acquire, renew, release)
+16. **Recovery** (state reconstruction)
+
+### 2.3 Service Dependencies
+
+Current services instantiated in kernel:
+
+```typescript
+private readonly runLeaseService: RunLeaseService;
+private readonly lockService: LockService;
+readonly collisionQueueService: CollisionQueueService;
+private readonly reportingService: ReportingService;
+private readonly featureStateService: FeatureStateService;
+private readonly featureLifecycleService: FeatureLifecycleService;
+private readonly planService: PlanService;
+private readonly patchService: PatchService;
+private readonly gateService: GateService;
+private readonly qaIndexService: QaIndexService;
+private readonly mergeService: MergeService;
+private readonly featureDeletionService: FeatureDeletionService;
+private readonly costTrackingService: CostTrackingService;
+private readonly performanceAnalyticsService: PerformanceAnalyticsService;
+private readonly gateSelectionService: GateSelectionService;
+```
+
+**Issue:** Kernel implements port interfaces for all these services, creating tight coupling.
+
+---
+
+## 3. Refactoring Strategy
+
+### 3.1 Phased Approach
+
+**Phase 1: Extract Configuration Management** (Week 1)
+
+- Create `ConfigurationService`
+- Move policy/gates/agents loading
+- ~200 lines extracted
+
+**Phase 2: Extract Core Repositories** (Week 1)
+
+- Create `IndexRepository`
+- Create `StateRepository`
+- Move file I/O and locking
+- ~300 lines extracted
+
+**Phase 3: Simplify Tool Routing** (Week 2)
+
+- Reduce kernel to thin delegation layer
+- All 67 methods become 1-2 line delegations
+- ~400 lines simplified
+
+**Phase 4: Extract Utility Functions** (Week 2)
+
+- Move helper functions to utility modules
+- ~100 lines extracted
+
+**Phase 5: Final Cleanup** (Week 2)
+
+- Remove redundant code
+- Optimize imports
+- ~200 lines removed
+
+### 3.2 Target Architecture
+
+```
+AopKernel (< 500 lines)
+├── ConfigurationService (policy, gates, agents, adapters)
+├── IndexRepository (index read/write, atomic updates)
+├── StateRepository (state read/write, locks)
+├── SchemaRegistry (validation)
+├── ToolRouter (dispatch to services)
+└── Services (existing 15 services)
+```
+
+---
+
+## 4. Detailed Refactoring Plan
+
+### 4.1 Phase 1: Extract Configuration Management
+
+#### 4.1.1 Create ConfigurationService
+
+**File:** `apps/control-plane/src/core/configuration-service.ts`
+
+**Responsibilities:**
+
+- Load and merge policy.yaml with defaults
+- Load gates.yaml
+- Load agents.yaml
+- Load adapters.yaml
+- Validate all configurations
+- Provide immutable snapshots
+
+**Interface:**
+
+```typescript
+export interface ConfigurationServicePort {
+  repoRoot: string;
+  schemaRegistry: SchemaRegistry;
+  configOverrides: KernelConfigOverrides;
+}
+
+export class ConfigurationService {
+  constructor(port: ConfigurationServicePort);
+
+  async loadAll(): Promise<void>;
+  getPolicySnapshot(): PolicyConfigSnapshot;
+  getGatesConfig(): AnyRecord;
+  getAgentsConfig(): AgentsConfigSnapshot;
+  getAdaptersConfig(): Record<string, unknown>;
+  isLoaded(): boolean;
+}
+```
+
+**Lines extracted:** ~200 lines from kernel.load()
+
+#### 4.1.2 Update Kernel
+
+Remove from kernel:
+
+- Policy loading logic
+- Gates loading logic
+- Agents loading logic
+- Adapters loading logic
+
+Add to kernel:
+
+```typescript
+private readonly configService: ConfigurationService;
+
+async load() {
+  await this.configService.loadAll();
+  this.policy = this.configService.getPolicySnapshot();
+  this.gatesConfig = this.configService.getGatesConfig();
+  this.agentsConfig = this.configService.getAgentsConfig();
+  this.adaptersConfig = this.configService.getAdaptersConfig();
+  this.loaded = true;
+}
+```
+
+**Benefit:** Kernel.load() reduced from ~90 lines to ~10 lines
+
+### 4.2 Phase 2: Extract Core Repositories
+
+#### 4.2.1 Create IndexRepository
+
+**File:** `apps/control-plane/src/infrastructure/index-repository.ts`
+
+**Responsibilities:**
+
+- Read/write index.json atomically
+- Provide index locking
+- Handle index schema validation
+- Manage index file path
+
+**Interface:**
+
+```typescript
+export interface IndexRepositoryPort {
+  repoRoot: string;
+  instanceId: string;
+  schemaRegistry: SchemaRegistry;
+}
+
+export class IndexRepository {
+  constructor(port: IndexRepositoryPort);
+
+  async read(): Promise<AnyRecord>;
+  async write(index: AnyRecord): Promise<void>;
+  async withLock<T>(operation: () => Promise<T>): Promise<T>;
+  getPath(): string;
+}
+```
+
+**Lines extracted:** ~150 lines
+
+**Methods moved from kernel:**
+
+- `readIndex()`
+- `writeIndex()`
+- `withIndexLock()`
+- Index file path logic
+
+#### 4.2.2 Create StateRepository
+
+**File:** `apps/control-plane/src/infrastructure/state-repository.ts`
+
+**Responsibilities:**
+
+- Read/write state.md files
+- Parse YAML frontmatter
+- Provide feature-level locking
+- Handle state schema validation
+
+**Interface:**
+
+```typescript
+export interface StateRepositoryPort {
+  repoRoot: string;
+  schemaRegistry: SchemaRegistry;
+}
+
+export class StateRepository {
+  constructor(port: StateRepositoryPort);
+
+  async read(featureId: string): Promise<{ frontMatter: AnyRecord; body: string }>;
+  async write(featureId: string, frontMatter: AnyRecord, body: string): Promise<void>;
+  async withLock<T>(featureId: string, operation: () => Promise<T>): Promise<T>;
+  getPath(featureId: string): string;
+}
+```
+
+**Lines extracted:** ~150 lines
+
+**Methods moved from kernel:**
+
+- `readState()`
+- `writeState()`
+- `withFeatureLock()`
+- State file path logic
+- YAML parsing logic
+
+#### 4.2.3 Update Services
+
+All services currently depend on kernel port interfaces. Update them to depend on repositories:
+
+**Before:**
+
+```typescript
+export interface PlanServicePort {
+  readIndex(): Promise<AnyRecord>;
+  writeIndex(index: AnyRecord): Promise<void>;
+  withIndexLock<T>(operation: () => Promise<T>): Promise<T>;
+  readState(featureId: string): Promise<StateReadResult>;
+  // ... 10 more methods
+}
+```
+
+**After:**
+
+```typescript
+export interface PlanServicePort {
+  indexRepository: IndexRepository;
+  stateRepository: StateRepository;
+  getPolicySnapshot(): AnyRecord;
+  planPath(featureId: string): string;
+  // ... only domain-specific methods
+}
+```
+
+**Benefit:** Port interfaces reduced by 50%, clearer separation of concerns
+
+---
+
+### 4.3 Phase 3: Simplify Tool Routing
+
+#### 4.3.1 Current State
+
+Kernel has 67 tool methods that look like:
+
+```typescript
+async featureInit(featureId) {
+  return await this.featureLifecycleService.featureInit(featureId);
+}
+
+async planSubmit(featureId, plan, expectedVersion) {
+  return await this.planService.planSubmit(featureId, plan, expectedVersion);
+}
+
+async gatesRun(featureId, profileName, mode) {
+  return await this.gateService.gatesRun(featureId, profileName, mode);
+}
+```
+
+**Issue:** 67 methods × 3 lines = 200+ lines of pure delegation
+
+#### 4.3.2 Refactored Approach
+
+**Option A: Direct Service Access (Recommended)**
+
+Remove delegation methods entirely. Update ToolRouter to call services directly:
+
+```typescript
+// In ToolRouter
+async executeTool(toolName: string, args: AnyRecord, context: ToolContext) {
+  const handler = this.handlers.get(toolName);
+  if (!handler) {
+    throw new Error(`Unknown tool: ${toolName}`);
+  }
+  return await handler.execute(args, context);
+}
+
+// Tool handlers access services directly
+class PlanSubmitHandler implements ToolHandler {
+  constructor(private planService: PlanService) {}
+
+  async execute(args: AnyRecord, context: ToolContext) {
+    return await this.planService.planSubmit(
+      args.feature_id,
+      args.plan_json,
+      args.expected_version
+    );
+  }
+}
+```
+
+**Benefit:** Eliminates 200+ lines of delegation code
+
+**Option B: Service Registry**
+
+Keep kernel as service locator:
+
+```typescript
+class AopKernel {
+  getService<T>(serviceName: string): T {
+    return this.services.get(serviceName) as T;
+  }
+}
+
+// Tool handlers get services from kernel
+const planService = kernel.getService<PlanService>('plan');
+```
+
+**Recommendation:** Use Option A for cleaner architecture
+
+#### 4.3.3 Update Tool Handlers
+
+Move tool handlers from inline kernel methods to dedicated handler classes:
+
+**File:** `apps/control-plane/src/application/tools/handlers/`
+
+```
+handlers/
+├── feature-init-handler.ts
+├── plan-submit-handler.ts
+├── gates-run-handler.ts
+├── locks-acquire-handler.ts
+├── merge-ready-handler.ts
+└── ... (33 total handlers)
+```
+
+Each handler:
+
+- Implements `ToolHandler` interface
+- Depends on specific services it needs
+- Contains tool-specific validation and orchestration
+
+**Lines extracted:** ~400 lines of tool orchestration logic
+
+---
+
+### 4.4 Phase 4: Extract Utility Functions
+
+#### 4.4.1 Create Utility Modules
+
+**File:** `apps/control-plane/src/core/utils/path-utils.ts`
+
+```typescript
+export function normalizeRepoPathForState(repoRoot: string, absolutePath: string): string;
+export function normalizeFromWorktree(
+  repoRoot: string,
+  worktreePath: string,
+  relativePath: string,
+): string;
+```
+
+**File:** `apps/control-plane/src/core/utils/field-readers.ts`
+
+```typescript
+export function readStringField(record: AnyRecord, key: string): string | null;
+export function readNumberField(record: AnyRecord, key: string): number | null;
+export function readBooleanField(record: AnyRecord, key: string): boolean | null;
+export function readObjectField(record: AnyRecord, key: string): AnyRecord;
+export function asArray<T>(value: unknown): T[];
+export function normalizeSet(array: string[]): string[];
+```
+
+**Lines extracted:** ~100 lines
+
+---
+
+### 4.5 Phase 5: Final Kernel Structure
+
+#### 4.5.1 Target Kernel Class
+
+**File:** `apps/control-plane/src/core/kernel.ts` (< 500 lines)
+
+```typescript
+export class AopKernel {
+  // Core dependencies
+  repoRoot: string;
+  instanceId: string;
+
+  // Infrastructure
+  private readonly configService: ConfigurationService;
+  private readonly indexRepository: IndexRepository;
+  private readonly stateRepository: StateRepository;
+  schemaRegistry: SchemaRegistry;
+
+  // Tool infrastructure
+  toolRegistry: AnyRecord | null;
+  toolHandlers: ToolHandlerRegistry;
+  toolRouter: ToolRouter;
+
+  // Services (unchanged)
+  private readonly runLeaseService: RunLeaseService;
+  private readonly lockService: LockService;
+  // ... (15 services)
+
+  // Public API
+  constructor(repoRoot: string, instanceId: string, configOverrides: KernelConfigOverrides);
+  async load(): Promise<void>;
+  async ensureLoaded(): Promise<void>;
+  async invoke(toolName: string, args: AnyRecord, context: ToolContext): Promise<ToolResponse>;
+
+  // Configuration access
+  getPolicySnapshot(): PolicyConfigSnapshot;
+  getGatesConfig(): AnyRecord;
+  getAgentsConfig(): AgentsConfigSnapshot;
+
+  // Repository access (for services)
+  getIndexRepository(): IndexRepository;
+  getStateRepository(): StateRepository;
+
+  // Provider management
+  setProvider(provider: WorkerProvider): void;
+  getProvider(): WorkerProvider | null;
+
+  // Recovery
+  async recoverFromState(): Promise<RecoveryResult>;
+}
+```
+
+**Total lines:** ~450 lines
+
+**Breakdown:**
+
+- Imports and types: ~50 lines
+- Constructor and initialization: ~100 lines
+- Service instantiation: ~100 lines
+- Public API methods: ~50 lines
+- Configuration/repository access: ~50 lines
+- Recovery logic: ~100 lines
+
+---
+
+## 5. Implementation Plan
+
+### 5.1 Week 1: Configuration and Repositories
+
+**Day 1-2: ConfigurationService**
+
+- Create `ConfigurationService` class
+- Extract policy loading logic
+- Extract gates/agents/adapters loading
+- Add tests
+- Update kernel to use ConfigurationService
+
+**Day 3-4: IndexRepository**
+
+- Create `IndexRepository` class
+- Extract index read/write/lock logic
+- Add tests
+- Update services to use IndexRepository
+
+**Day 5: StateRepository**
+
+- Create `StateRepository` class
+- Extract state read/write/lock logic
+- Add tests
+- Update services to use StateRepository
+
+**Verification:**
+
+- All tests pass
+- Kernel reduced to ~1200 lines
+- No behavior changes
+
+### 5.2 Week 2: Tool Routing and Cleanup
+
+**Day 1-2: Tool Handler Extraction**
+
+- Create handler classes for all 33 tools
+- Update ToolRouter to use handlers
+- Remove delegation methods from kernel
+
+**Day 3: Utility Extraction**
+
+- Move utility functions to separate modules
+- Update imports across codebase
+
+**Day 4: Service Port Updates**
+
+- Update all service port interfaces
+- Remove redundant methods
+- Simplify dependencies
+
+**Day 5: Final Cleanup and Verification**
+
+- Remove dead code
+- Optimize imports
+- Run full test suite
+- Performance benchmarks
+- Documentation updates
+
+**Verification:**
+
+- Kernel < 500 lines
+- All tests pass
+- Performance within ±5%
+- Zero lint warnings
+
+---
+
+## 6. Testing Strategy
+
+### 6.1 Unit Tests
+
+**New Tests Required:**
+
+1. **ConfigurationService**
+   - Policy loading and merging
+   - Gates/agents/adapters loading
+   - Validation failures
+   - Config overrides
+
+2. **IndexRepository**
+   - Atomic read/write
+   - Lock acquisition
+   - Concurrent access
+   - Schema validation
+
+3. **StateRepository**
+   - YAML parsing
+   - Feature-level locking
+   - Concurrent updates
+   - Version conflicts
+
+4. **Tool Handlers**
+   - Each handler's execute method
+   - Argument validation
+   - Error handling
+   - Service orchestration
+
+**Coverage Target:** ≥90% for all new classes
+
+### 6.2 Integration Tests
+
+**Existing Tests Must Pass:**
+
+- All kernel tests in `apps/control-plane/test/kernel.spec.ts`
+- All tool runtime tests
+- All supervisor tests
+- All CLI tests
+
+**New Integration Tests:**
+
+- End-to-end tool execution through new architecture
+- Multi-feature parallel execution
+- Recovery scenarios
+- Lock contention scenarios
+
+### 6.3 Performance Tests
+
+**Benchmarks:**
+
+- Tool invocation latency (must be within ±5%)
+- Feature initialization time
+- Gate execution time
+- Index/state read/write time
+
+**Load Tests:**
+
+- 5 features in parallel
+- 100 tool invocations
+- Lock contention under load
+
+---
+
+## 7. Migration Path
+
+### 7.1 Backward Compatibility
+
+**Guaranteed:**
+
+- All tool contracts unchanged
+- All error codes unchanged
+- All CLI commands unchanged
+- All file formats unchanged
+- All schemas unchanged
+
+**Internal Changes Only:**
+
+- Service instantiation
+- Port interfaces
+- Internal method signatures
+
+### 7.2 Rollback Plan
+
+**If Issues Arise:**
+
+1. **Immediate:** Revert to previous kernel implementation
+2. **Short-term:** Fix issues in new architecture
+3. **Long-term:** Complete migration with fixes
+
+**Rollback Triggers:**
+
+- Test failures >5%
+- Performance degradation >10%
+- Production incidents
+- Unresolvable bugs
+
+---
+
+## 8. Success Metrics
+
+### 8.1 Quantitative Metrics
+
+- ✅ Kernel lines: 1768 → <500 (70% reduction)
+- ✅ Cyclomatic complexity: Reduced by 60%
+- ✅ Service coupling: Reduced by 50%
+- ✅ Test coverage: Maintained at ≥90%
+- ✅ Performance: Within ±5% of baseline
+
+### 8.2 Qualitative Metrics
+
+- ✅ Code readability improved
+- ✅ Service responsibilities clear
+- ✅ Easier to add new tools
+- ✅ Easier to modify services
+- ✅ Better error messages
+- ✅ Improved documentation
+
+---
+
+## 9. Risks and Mitigation
+
+### 9.1 Risks
+
+**High Risk:**
+
+- Breaking existing tool contracts
+- Performance degradation
+- Test failures
+
+**Medium Risk:**
+
+- Increased complexity in service wiring
+- Circular dependencies between services
+- Configuration loading order issues
+
+**Low Risk:**
+
+- Documentation gaps
+- Migration time overruns
+
+### 9.2 Mitigation Strategies
+
+**For Breaking Changes:**
+
+- Comprehensive integration tests before merge
+- Staged rollout with feature flags
+- Immediate rollback capability
+
+**For Performance:**
+
+- Benchmark before/after
+- Profile hot paths
+- Optimize critical sections
+
+**For Complexity:**
+
+- Clear dependency graphs
+- Explicit initialization order
+- Dependency injection container
+
+---
+
+## 10. Acceptance Criteria
+
+### 10.1 Must Pass
+
+- [ ] Kernel class <500 lines
+- [ ] All 67 tool methods delegated or removed
+- [ ] ConfigurationService extracted and tested
+- [ ] IndexRepository extracted and tested
+- [ ] StateRepository extracted and tested
+- [ ] All tool handlers extracted
+- [ ] All utility functions extracted
+- [ ] All existing tests pass
+- [ ] `npm run typecheck` passes
+- [ ] `npm run lint` passes with 0 warnings
+- [ ] Performance within ±5% of baseline
+- [ ] Test coverage ≥90%
+
+### 10.2 Documentation
+
+- [ ] Architecture diagram updated
+- [ ] Service responsibilities documented
+- [ ] Port interfaces documented
+- [ ] Migration guide created
+- [ ] CLAUDE.md updated
+- [ ] README.md updated
+
+---
+
+## 11. References
+
+### 11.1 Related Documents
+
+- `CLAUDE.md` - Architecture patterns and principles
+- `ARCHITECTURE_ADHERENCE_ANALYSIS.md` - Current state analysis
+- `spec-files/completed/agentic_orchestrator_oop_refactor_spec.md` - OOP refactor spec
+- `prompts/agentic-documentation-guide.instructions.md` - Documentation standards
+
+### 11.2 Related Code
+
+- `apps/control-plane/src/core/kernel.ts` - Current kernel (1768 lines)
+- `apps/control-plane/src/application/services/` - Existing services (24 files)
+- `apps/control-plane/src/application/tools/` - Tool infrastructure
+- `apps/control-plane/test/kernel.spec.ts` - Kernel tests
+
+---
+
+## 12. Appendix
+
+### 12.1 Current Kernel Method Inventory
+
+**Configuration (5 methods):**
+
+- `load()`, `ensureLoaded()`, `getPolicySnapshot()`, `getGatesConfig()`, `getAgentsConfig()`
+
+**Tool Routing (67 methods):**
+
+- All tool delegation methods (feature._, plan._, gates._, locks._, etc.)
+
+**Infrastructure (10 methods):**
+
+- `readIndex()`, `writeIndex()`, `withIndexLock()`
+- `readState()`, `writeState()`, `withFeatureLock()`
+- `validateSchema()`, `readRunLease()`, `writeRunLease()`
+
+**Utility (8 methods):**
+
+- Path normalization, field readers, array helpers
+
+**Total:** 90 methods
+
+### 12.2 Target Kernel Method Inventory
+
+**Public API (10 methods):**
+
+- `constructor()`, `load()`, `ensureLoaded()`, `invoke()`
+- `getPolicySnapshot()`, `getGatesConfig()`, `getAgentsConfig()`
+- `getIndexRepository()`, `getStateRepository()`
+- `setProvider()`, `getProvider()`
+
+**Internal (5 methods):**
+
+- Service instantiation helpers
+- Recovery logic
+
+**Total:** 15 methods (83% reduction)
+
+---
+
+**End of Specification**