agentic-orchestrator 0.1.13 → 0.1.15

package/spec-files/outstanding/agentic_orchestrator_real_worker_provider_execution_spec.md

@@ -0,0 +1,616 @@
+# Feature Spec: Real Worker Provider Execution Path and Cluster Role Semantics (AOP)
+
+> **Purpose of this document**: Define implementation-ready changes to replace stubbed worker execution (`NullWorkerProvider`) in `aop run`/`aop resume` with real provider-backed execution, while preserving deterministic orchestration and making planner/builder/qa behavior explicit and auditable.
+
+**Version:** 1.1  
+**Date:** 2026-03-04  
+**Status:** Draft  
+**Roadmap Mapping:** M42
+
+---
+
+## 0. Standards and Dependencies
+
+### 0.1 Required Standards
+
+All implementation MUST preserve:
+
+- deterministic tool envelopes and error normalization
+- MCP/in-process transport parity at the tool boundary
+- strict schema validation (`validate:mcp-contracts`, `validate:architecture`)
+- canonical state under `.aop/**` as source of truth
+- run lease + session ownership invariants from global orchestrator topology spec
+
+### 0.2 Required Upstream Inputs
+
+Implementing agents MUST read:
+
+- `spec-files/completed/agentic_orchestrator_single_global_orchestrator_spec.md`
+- `spec-files/outstanding/agentic_orchestrator_provider_auth_bootstrap_spec.md`
+- `apps/control-plane/src/cli/run-command-handler.ts`
+- `apps/control-plane/src/cli/resume-command-handler.ts`
+- `apps/control-plane/src/providers/providers.ts`
+- `apps/control-plane/src/supervisor/runtime.ts`
+- `apps/control-plane/src/supervisor/session-orchestrator.ts`
+- `apps/control-plane/src/supervisor/planning-wave-executor.ts`
+- `apps/control-plane/src/supervisor/build-wave-executor.ts`
+- `apps/control-plane/src/supervisor/qa-wave-executor.ts`
+- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `agentic/orchestrator/agents.yaml`
+- `agentic/orchestrator/schemas/agents.schema.json`
+- `agentic/orchestrator/policy.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+
+### 0.3 Scope
+
+This spec implements:
+
+- real provider-backed worker execution for `run` and `resume`
+- provider factory/wiring so selected provider determines runtime behavior
+- strict planner/builder/qa role contracts for outputs and progress
+- deterministic handling of no-output/no-progress/malformed-output failure modes
+- regression coverage so `npm run verify` catches the current “plan -> ready_to_merge without edits” defect class
+
+Out of scope:
+
+- new model vendors beyond currently registered provider names
+- replacing gate engine semantics
+- changing merge approval governance rules
+
+### 0.4 OOP and Architecture Guardrails (Mandatory)
+
+Implementation for this spec MUST follow explicit object-oriented and architectural best practices:
+
+- **KISS**: prefer the simplest design that satisfies deterministic behavior and contract requirements; avoid speculative abstractions.
+- **SRP (Single Responsibility Principle)**: each class/module should have one clear reason to change (factory selection, provider execution, parsing, orchestration policy, etc.).
+- **SOLID-oriented design** (applied pragmatically):
+  - open/closed for new provider adapters without modifying orchestration flow logic
+  - interface segregation for provider capabilities (`WorkerProvider`, parser contracts, factory contracts)
+  - dependency inversion via ports/interfaces rather than direct runtime coupling
+- **Hexagonal architecture alignment**: keep provider SDK/CLI process details in adapter layer; supervisor/core should depend on ports/contracts only.
+- **Composition over inheritance**: adapters compose parser/runner dependencies; avoid deep class hierarchies.
+- **High cohesion / low coupling**: no mixing of CLI parsing, provider transport logic, and supervisor decision routing in the same class.
+
+Code organization requirements for this spec:
+
+- factory logic remains in dedicated factory module(s)
+- provider transport concerns remain in adapter modules
+- provider payload normalization remains in parser modules
+- supervisor wave executors and decision loop consume normalized contracts only
+
+---
+
+## 1. Architectural Critique of Current Behavior
+
+1. **`run` and `resume` always instantiate `NullWorkerProvider`**
+
+- `resolveProviderSelection()` resolves provider/model, but the selected provider is not used to choose a runtime implementation.
+- Result: all providers execute a stub path.
+
+2. **`NullWorkerProvider.runWorker()` emits NOTE-only behavior**
+
+- Planner/builder/qa paths return `NOTE` instead of substantive worker actions (`PLAN_SUBMISSION`, `PATCH`, actionable `REQUEST` flows).
+- Result: no code modification signal can exist unless externally injected.
+
+3. **Planner fallback auto-submits a generated plan**
+
+- `PlanningWaveExecutor` submits deterministic initial plan when no plan is returned by worker.
+- Result: planning phase appears to complete even when provider produced no meaningful work.
+
+4. **Promotion is status/gate-driven, not delivery-driven**
+
+- Build/QA phases can pass gates and transition state even when no patches occurred.
+- Result: feature can reach `ready_to_merge` with empty worktree diff.
+
+5. **Cluster sessions give false confidence**
+
+- Session IDs are created/rotated and persisted, so runtime looks healthy in dashboard/index.
+- Result: “session active” logs hide execution no-op.
+
+6. **Current tests do not hard-fail on stub execution in run/resume**
+
+- Existing coverage validates many orchestration mechanics but not “live provider required for delivery” invariants.
+- Result: regression can pass `verify` while actual feature delivery is broken.
+
+---
+
+## 2. Revised Architecture Fit
+
+### 2.1 Provider Runtime Selection Must Be Explicit
+
+Introduce a provider runtime factory used by `run`, `resume`, and attach/send paths:
+
+- `createWorkerProvider(selection, context)` returns concrete provider implementation
+- `run`/`resume` MUST require `mode=live` by default
+- stub mode remains allowed only for tests/explicit local override
+
+### 2.2 Worker Execution Is a Contracted Output Channel
+
+`WorkerProvider.runWorker(...)` MUST return normalized output envelopes consumed by `WorkerDecisionLoop`:
+
+- supported output types: `PLAN_SUBMISSION`, `PATCH`, `REQUEST`, `NOTE`
+- malformed output is deterministic error, not silent ignore
+- multi-output responses MUST preserve deterministic order
+
+### 2.3 Role Semantics Across Cluster
+
+Planner, builder, and QA roles keep their current tool firewall, but output expectations become explicit:
+
+- planner: must drive plan evolution (`PLAN_SUBMISSION` / `amend_plan` request path)
+- builder: must propose patch actions before/around fast gate retries
+- qa: must propose remediation patches before/around full gate retries
+- note-only execution is allowed transiently, not indefinitely
+
+### 2.4 Session Topology Compatibility
+
+No change to `1 + 3N` topology:
+
+- one orchestrator session per run
+- planner/builder/qa per active feature
+- queued features remain `unassigned`
+- QA session rotation remains supported and must work with real provider sessions
+
+---
+
+## 3. Contract and Schema Deltas
+
+### 3.1 Agents Runtime Contract (`agents.schema.json`)
+
+Add runtime execution controls:
+
+```yaml
+runtime:
+  worker_provider_mode: live # live | stub
+  worker_response_timeout_ms: 120000
+  max_consecutive_no_progress_iterations: 2
+```
+
+Rules:
+
+- `worker_provider_mode=live` is default for `aop run` and `aop resume`
+- `stub` is permitted only in tests or explicit debug mode
+- timeout and no-progress thresholds must be bounded positive integers
+
+### 3.2 Policy Contract (`policy.schema.json`)
+
+Add deterministic enforcement toggles:
+
+```yaml
+execution:
+  require_live_provider_for_run: true
+  malformed_worker_output_action: block_feature # block_feature | fail_run
+  no_progress_action: block_feature # block_feature | fail_run
+```
+
+Rules:
+
+- when `require_live_provider_for_run=true`, constructing a stub provider in run/resume is a hard error
+- action routing is deterministic and auditable in decisions log
+
+### 3.3 Worker Output Envelope Contract
+
+Normalize provider output:
+
+```json
+{
+  "session_id": "builder-feature_x-...",
+  "role": "builder",
+  "feature_id": "feature_x",
+  "outputs": [
+    { "type": "PATCH", "unified_diff": "..." },
+    { "type": "NOTE", "content": "Applied auth fix for retry #1" }
+  ],
+  "provider_meta": {
+    "provider": "claude",
+    "model": "local-default"
+  }
+}
+```
+
+Rules:
+
+- `outputs` is required; empty outputs count as no-progress
+- unknown output `type` is invalid output
+- provider adapter is responsible for parsing provider-native payload into this envelope
+
+### 3.4 Runtime Diagnostic Artifact Contract
+
+Add worker execution journal:
+
+```text
+.aop/runtime/worker-events/<run_id>.jsonl
+```
+
+Each entry:
+
+```json
+{
+  "ts": "2026-03-04T00:00:00.000Z",
+  "run_id": "run:...",
+  "feature_id": "feature_x",
+  "role": "builder",
+  "session_id": "builder-feature_x-...",
+  "output_types": ["PATCH", "NOTE"],
+  "patch_count": 1,
+  "plan_submission_count": 0,
+  "request_count": 0,
+  "note_count": 1,
+  "valid": true,
+  "error_code": null
+}
+```
+
+Rules:
+
+- append-only JSONL
+- used for operator diagnostics and regression assertions
+
+### 3.5 Error Taxonomy Extensions
+
+Add deterministic error codes:
+
+- `provider_stub_disallowed`
+- `provider_runtime_unavailable`
+- `provider_output_invalid`
+- `provider_no_progress`
+
+---
+
+## 4. Provider Runtime Design
+
+### 4.1 ProviderFactory Interface (Normative)
+
+The provider module MUST expose a concrete factory contract, not just helper functions:
+
+```ts
+export type WorkerProviderMode = 'live' | 'stub';
+export type ProviderCommandContext = 'run' | 'resume' | 'send' | 'attach';
+
+export interface CreateWorkerProviderInput {
+  selection: ProviderSelection;
+  mode: WorkerProviderMode;
+  context: ProviderCommandContext;
+  policy: {
+    require_live_provider_for_run: boolean;
+  };
+  runtime: {
+    worker_response_timeout_ms: number;
+    max_consecutive_no_progress_iterations: number;
+  };
+  commandRunner?: ProviderCommandRunner;
+}
+
+export interface WorkerProviderFactory {
+  create(input: CreateWorkerProviderInput): WorkerProvider;
+}
+```
+
+Requirements:
+
+- `run` and `resume` MUST call `factory.create(...)` exactly once per CLI invocation.
+- the returned `WorkerProvider` instance is shared by planner/builder/qa/orchestrator session management for that runtime.
+- `create(...)` is the single enforcement point for `provider_stub_disallowed`.
+
+### 4.2 Concrete Objects the Factory Must Create
+
+The spec requires an explicit object set so planner/builder/qa behavior is not ambiguous:
+
+1. `DefaultWorkerProviderFactory` (new)
+
+- owns provider-to-adapter mapping and mode/policy enforcement
+- created in CLI composition (`bootstrap.ts`) and injected into run/resume handlers
+
+2. `NullWorkerProvider` (existing)
+
+- only returned when `mode='stub'` and policy/context allows it
+
+3. `CliWorkerProvider` (new live adapter)
+
+- handles CLI-backed providers: `codex`, `claude`, `kiro-cli`, `copilot`, `custom`
+- responsible for command invocation + payload parsing + normalized envelope output
+
+4. `ApiWorkerProvider` (new live adapter)
+
+- handles API-backed providers (initially `gemini`)
+- responsible for SDK/API request/response + normalized envelope output
+
+5. `ProviderOutputParser` implementations (new)
+
+- `CodexOutputParser`
+- `ClaudeOutputParser`
+- `GenericCliOutputParser` (`kiro-cli`/`copilot`/`custom` initial path)
+- `GeminiOutputParser`
+
+The live provider adapters MUST compose a parser object; parsing logic MUST NOT live in wave executors.
+
+### 4.3 Provider Mapping Table (Normative)
+
+`DefaultWorkerProviderFactory` MUST use deterministic mapping:
+
+| `selection.provider`   | `mode` | Adapter object                                   |
+| ---------------------- | ------ | ------------------------------------------------ |
+| `codex`                | `live` | `CliWorkerProvider` + `CodexOutputParser`        |
+| `claude`               | `live` | `CliWorkerProvider` + `ClaudeOutputParser`       |
+| `kiro-cli`             | `live` | `CliWorkerProvider` + `GenericCliOutputParser`   |
+| `copilot`              | `live` | `CliWorkerProvider` + `GenericCliOutputParser`   |
+| `custom`               | `live` | `CliWorkerProvider` + `GenericCliOutputParser`   |
+| `gemini`               | `live` | `ApiWorkerProvider` + `GeminiOutputParser`       |
+| any supported provider | `stub` | `NullWorkerProvider` (subject to policy/context) |
+
+Unknown provider names are already rejected by `resolveProviderSelection()` and MUST NOT reach factory mapping.
+
+### 4.4 Factory Decision Algorithm (Normative)
+
+`DefaultWorkerProviderFactory.create(input)` must behave as follows:
+
+1. Validate provider is supported.
+2. If `input.mode === 'stub'`:
+
+- if `context in {'run','resume'}` and `policy.require_live_provider_for_run === true`, throw `provider_stub_disallowed`.
+- else return `new NullWorkerProvider(selection, ...)`.
+
+3. If `input.mode === 'live'`:
+
+- resolve adapter class by mapping table.
+- instantiate adapter with deterministic dependencies (`selection`, timeout, command runner/api client, parser).
+- if required runtime dependency is unavailable, throw `provider_runtime_unavailable`.
+
+4. Return `WorkerProvider`.
+
+### 4.5 Planner/Builder/QA Provider Usage and Object Lifecycle
+
+This section defines how cluster roles actually consume the factory output:
+
+1. CLI bootstrap resolves selection and creates a single `WorkerProvider` instance via factory.
+2. `SupervisorRuntime` receives that single provider instance.
+3. `SessionOrchestrator` creates role sessions through this provider:
+
+- orchestrator: `createSession('orchestrator','global',null)`
+- planner: `createSession('planner', featureId, prompts.planner)`
+- builder: `createSession('builder', featureId, prompts.builder)`
+- qa: `createSession('qa', featureId, prompts.qa)`
+
+4. `WorkerDecisionLoop` invokes `provider.runWorker(...)` for planner/builder/qa turns.
+5. QA rotation closes prior QA session and re-creates QA session through the same provider instance.
+
+Important clarification: planner/builder/qa do not each instantiate providers directly; they consume sessions from the runtime-scoped provider instance created by `ProviderFactory`.
+
+### 4.6 Live Adapter Responsibilities
+
+Each live adapter MUST:
+
+- create/reattach/close sessions deterministically
+- submit role instructions + context bundle to provider runtime
+- parse provider response into normalized `outputs`
+- surface transport/process failures via normalized error codes
+- emit provider metadata needed for worker-events journaling
+
+### 4.7 Stub Adapter Responsibilities
+
+`NullWorkerProvider` stays in codebase for:
+
+- unit tests
+- explicit debug mode
+
+It MUST NOT be default for production `run`/`resume`.
+
+### 4.8 File Decomposition Requirement for ProviderFactory
+
+Implementation must split provider concerns into explicit files (exact names may vary but boundaries are mandatory):
+
+- factory: `providers/worker-provider-factory.ts`
+- adapter(s): `providers/cli-worker-provider.ts`, `providers/api-worker-provider.ts`
+- parser(s): `providers/output-parsers/*.ts`
+- compatibility export barrel: `providers/providers.ts`
+
+`providers/providers.ts` may re-export these modules for compatibility, but factory logic MUST NOT remain implicit inside CLI handlers.
+
+---
+
+## 5. Cluster Agent Behavior Changes
+
+### 5.1 Planner Agent
+
+Current behavior to change:
+
+- planner can silently rely on generated fallback plan.
+
+Required behavior:
+
+- in live mode, if planner returns no valid plan actions for `N` iterations, feature is blocked with `provider_no_progress`
+- fallback initial plan generation remains allowed only when provider mode is `stub` (or explicit policy override)
+
+### 5.2 Builder Agent
+
+Current behavior to change:
+
+- builder can run fast gates after NOTE-only outputs.
+
+Required behavior:
+
+- builder may emit NOTE/REQUEST, but repeated no-progress cycles in live mode block feature
+- gate-repair retries still supported, but retries must include a valid worker output envelope each cycle
+
+### 5.3 QA Agent
+
+Current behavior to keep + harden:
+
+- QA session rotation after each cycle stays in place.
+
+Required behavior:
+
+- rotation must preserve continuity despite provider-backed sessions
+- malformed/no-progress QA outputs in live mode trigger deterministic block/fail action
+- full-gate retry loop remains, but must be backed by valid worker execution envelopes
+
+### 5.4 Orchestrator Agent
+
+No role contract expansion, but orchestrator must:
+
+- journal provider execution diagnostics
+- apply policy actions for malformed/no-progress outputs
+- preserve queue/session invariants for active vs queued features
+
+---
+
+## 6. Edge-Case Matrix (Normative)
+
+1. Provider binary/SDK unavailable at startup:
+
+- `run`/`resume` fails fast with `provider_runtime_unavailable`.
+
+2. Provider session creation succeeds but runWorker times out:
+
+- deterministic timeout error; action governed by policy (`block_feature` vs `fail_run`).
+
+3. Provider returns unparsable payload:
+
+- record invalid output event; apply malformed-output action.
+
+4. Provider returns unknown output types:
+
+- treat as invalid output (not ignored silently).
+
+5. Provider returns NOTE-only for repeated iterations:
+
+- treated as no-progress once threshold exceeded.
+
+6. Resume path with stale runtime sessions:
+
+- reattach when possible; otherwise create new sessions without violating `1 + 3N`.
+
+7. Takeover of stale run with epoch increment:
+
+- orphan sessions closed/marked; new live provider sessions established.
+
+8. Queued features:
+
+- remain `unassigned`; no worker provider calls until promoted active.
+
+9. QA session rotation failures:
+
+- close/create failures are deterministic provider runtime errors; no silent skip.
+
+10. Concurrent features with max parallel gates:
+
+- provider errors isolated per feature; other active features continue unless policy says `fail_run`.
+
+---
+
+## 7. Implementation Plan
+
+### M42.1 Provider Factory + Wiring
+
+- replace direct `new NullWorkerProvider(...)` in:
+  - `run-command-handler.ts`
+  - `resume-command-handler.ts`
+- add provider factory wiring in CLI bootstrap composition path
+- preserve attach/send behavior via factory output
+
+### M42.2 Live Provider Adapter Layer
+
+- split provider module into:
+  - selection + credentials
+  - provider factory
+  - live adapter(s)
+  - stub adapter
+- keep public exports backward-compatible where feasible
+
+### M42.3 Worker Decision Hardening
+
+- validate output envelope before routing decisions
+- record worker-events journal entries
+- enforce no-progress counters and policy action
+
+### M42.4 Planner/Builder/QA Execution Semantics
+
+- planner fallback constrained by provider mode/policy
+- builder/qa no-progress behavior enforced in live mode
+- QA rotation compatibility tested with live adapter stubs/fakes
+
+### M42.5 Regression and Verify Hardening
+
+- add targeted tests to ensure `npm run verify` fails when run/resume silently use stub mode
+- add integration test proving real provider outputs lead to plan + patch + gates progression
+
+---
+
+## 8. File-Level Change Targets
+
+- `apps/control-plane/src/providers/providers.ts`
+- `apps/control-plane/src/providers/worker-provider-factory.ts` (new)
+- `apps/control-plane/src/providers/cli-worker-provider.ts` (new)
+- `apps/control-plane/src/providers/api-worker-provider.ts` (new)
+- `apps/control-plane/src/providers/output-parsers/` (new folder)
+- `apps/control-plane/src/cli/run-command-handler.ts`
+- `apps/control-plane/src/cli/resume-command-handler.ts`
+- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
+- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
+- `apps/control-plane/src/supervisor/planning-wave-executor.ts`
+- `apps/control-plane/src/supervisor/build-wave-executor.ts`
+- `apps/control-plane/src/supervisor/qa-wave-executor.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `agentic/orchestrator/schemas/agents.schema.json`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/agents.yaml`
+- `agentic/orchestrator/policy.yaml`
+- `apps/control-plane/test/providers.spec.ts`
+- `apps/control-plane/test/cli.unit.spec.ts`
+- `apps/control-plane/test/resume-command.spec.ts`
+- `apps/control-plane/test/worker-decision-loop.spec.ts`
+- `apps/control-plane/test/supervisor.spec.ts`
+- `apps/control-plane/test/supervisor.unit.spec.ts`
+
+---
+
+## 9. Test Plan (Must Be in `npm run verify` Path)
+
+### 9.1 Unit
+
+- provider factory returns live adapter for run/resume defaults
+- stub mode requires explicit opt-in
+- malformed envelope parsing returns `provider_output_invalid`
+
+### 9.2 Supervisor Integration
+
+- GIVEN live provider emits plan+patch outputs WHEN run executes THEN feature transitions through planning/building/qa with non-empty diff evidence
+- GIVEN live provider emits repeated NOTE-only outputs WHEN threshold exceeded THEN feature blocks with `provider_no_progress`
+- GIVEN queued features WHEN runtime starts THEN no worker calls occur for queued features
+
+### 9.3 Resume/Recovery
+
+- GIVEN resumable feature WHEN resume executes THEN factory uses live adapter and respects existing session assignments
+- GIVEN stale sessions/timeouts WHEN resume executes THEN recovery and epoch/session invariants hold
+
+### 9.4 Regression for Current Defect
+
+- GIVEN run path configured for live mode WHEN provider resolves to stub THEN command fails (not silently proceeds)
+- GIVEN no patch and no valid worker progress in live mode WHEN gates pass spuriously THEN feature does not auto-promote to `ready_to_merge`
+
+---
+
+## 10. Acceptance Criteria
+
+1. `aop run` and `aop resume` do not use `NullWorkerProvider` unless explicit stub mode is enabled.
+2. Provider selection materially changes runtime behavior through factory-selected live adapters.
+3. Planner/builder/qa output processing is deterministic and validated.
+4. Malformed/no-progress worker behaviors are auditable and policy-controlled.
+5. Global topology invariants (`1 + 3N`, queued `unassigned`) remain intact.
+6. `npm run verify` contains regression tests that fail for the “planning -> ready_to_merge with no build output/diff” defect class.
+
+---
+
+## 11. Risks and Mitigations
+
+- Risk: provider-specific payload variability breaks parser.
+  - Mitigation: strict adapter normalization layer + invalid-output policy action.
+
+- Risk: session churn (especially QA rotation) increases provider flakiness.
+  - Mitigation: explicit timeout handling + deterministic recovery + targeted QA rotation tests.
+
+- Risk: introducing live enforcement breaks existing tests that rely on stub behavior.
+  - Mitigation: explicit stub mode for tests and incremental migration of unit fixtures.
+
+- Risk: false positives in no-progress detection for legitimately small/no-op work.
+  - Mitigation: policy threshold and explicit override path with audited rationale.