agentic-orchestrator 0.1.0

package/spec-files/agentic_orchestrator_dot_aop_generated_artifacts_spec.md

@@ -0,0 +1,211 @@
+# Feature Spec: Relocate Generated Runtime Artifacts to `.aop/` (AOP)
+
+> **Purpose of this document**: Define an implementation-ready migration plan that moves runtime-generated AOP files out of `agentic/` and into a dedicated hidden runtime directory `.aop/`, while preserving deterministic MCP/Supervisor behavior and existing CLI/tool semantics.
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- Runtime-generated artifacts MUST be written under `.aop/` instead of `agentic/`.
+- Deterministic orchestration guarantees MUST remain unchanged:
+  - schema validation
+  - optimistic concurrency/version checks
+  - lock/collision behavior
+  - gate/evidence behavior
+  - explicit merge control
+- CLI workflows (`run`, `status`, `resume`, `stop`) MUST keep existing behavior and error envelopes.
+- Existing repositories with generated files under `agentic/` MUST be supported through a deterministic migration/compatibility path.
+- Path ownership MUST be explicit:
+  - source-managed orchestrator config stays in `agentic/orchestrator/`
+  - runtime-generated feature/runtime state moves to `.aop/`
+
+### 1.2 Non-Goals
+
+- No changes to tool names, role permissions, or gate policy semantics.
+- No replacement of file-backed state with DB-backed persistence.
+- No provider/session model changes.
+
+---
+
+## 2. Canonical Path Contract
+
+### 2.1 Source-Managed (unchanged)
+
+- `agentic/orchestrator/**` remains source-managed:
+  - `gates.yaml`, `policy.yaml`, `agents.yaml`
+  - `schemas/**`
+  - `tools/catalog.json`, `tools/protocol.json`, `tools/errors.schema.json`, `tools/schemas/**`
+
+### 2.2 Runtime-Generated (new location)
+
+- `.aop/features/index.json`
+- `.aop/features/<feature_id>/spec.md` (canonical ingested spec copy)
+- `.aop/features/<feature_id>/state.md`
+- `.aop/features/<feature_id>/plan.json`
+- `.aop/features/<feature_id>/qa_test_index.json`
+- `.aop/features/<feature_id>/decisions.md`
+- `.aop/features/<feature_id>/logs/**`
+- `.aop/features/<feature_id>/evidence/**`
+- `.aop/runtime/operation-ledger/<run_id>.json`
+
+### 2.3 Discovery/Canonicalization Rules
+
+- `feature.discover_specs` MUST discover canonical specs under `.aop/features/*/spec.md`.
+- `aop run -fi/-fl` MUST ingest non-canonical input into `.aop/features/<feature_id>/spec.md`.
+- Feature ID derivation rules (`.spec`, `-spec`, slug regex) MUST remain unchanged.
+
+---
+
+## 3. Architecture Decisions
+
+### 3.1 Introduce a Single Runtime Path Authority
+
+- Add a typed path-layout abstraction (for example `AopPathLayout`) used by CLI, kernel, services, and MCP runtime helpers.
+- The abstraction MUST provide explicit getters for:
+  - source config roots (`agentic/orchestrator`)
+  - runtime data roots (`.aop/features`, `.aop/runtime`)
+  - per-feature generated files (`state.md`, `plan.json`, etc.)
+
+### 3.2 Keep Schema/Policy Roots Stable
+
+- Schema/policy/catalog load paths remain under `agentic/orchestrator/**`.
+- Only generated runtime paths move to `.aop/**`.
+
+### 3.3 Deterministic Compatibility Window
+
+- Startup/runtime access MUST support deterministic migration from legacy `agentic/features/**` and `agentic/runtime/**`:
+  - if `.aop/**` exists, it is authoritative
+  - if `.aop/**` is absent and legacy exists, migrate/copy forward
+  - no destructive deletion during migration step
+
+---
+
+## 4. Required Code Changes
+
+### 4.1 Core and Services
+
+- Update runtime path ownership in:
+  - `apps/control-plane/src/core/kernel.ts`
+  - `apps/control-plane/src/application/services/feature-lifecycle-service.ts`
+  - `apps/control-plane/src/application/services/feature-state-service.ts`
+  - `apps/control-plane/src/application/services/plan-service.ts`
+  - `apps/control-plane/src/application/services/qa-index-service.ts`
+  - `apps/control-plane/src/application/services/gate-service.ts`
+  - `apps/control-plane/src/application/services/reporting-service.ts`
+  - `apps/control-plane/src/application/services/merge-service.ts`
+- Ensure lock/index/state atomic-write paths now target `.aop/features/**`.
+
+### 4.2 CLI and Spec Ingestion
+
+- Update canonical spec detection/ingestion:
+  - `apps/control-plane/src/cli/spec-utils.ts`
+  - `apps/control-plane/src/cli/spec-input-resolver.ts`
+  - `apps/control-plane/src/cli/spec-ingestion-service.ts`
+- CLI outputs and status payload paths MUST reference `.aop/**` canonical locations.
+
+### 4.3 MCP Runtime Helpers
+
+- Move operation ledger storage root from `agentic/runtime/operation-ledger` to `.aop/runtime/operation-ledger` in:
+  - `apps/control-plane/src/mcp/operation-ledger.ts`
+
+### 4.4 Documentation and Ignore Rules
+
+- Update `README.md` runtime path references from `agentic/features/**` to `.aop/features/**` where describing generated artifacts.
+- Add `.aop/` to `.gitignore` (except intentional tracked placeholders if any).
+
+---
+
+## 5. Test Plan (Normative)
+
+### 5.1 Unit/Integration Coverage Additions
+
+- Add/adjust tests for:
+  - canonical path detection for `.aop/features/<id>/spec.md`
+  - non-canonical ingestion target path under `.aop/features/<id>/spec.md`
+  - `feature.discover_specs` returning `.aop/**` paths
+  - operation-ledger writes to `.aop/runtime/operation-ledger/**`
+  - state/index/QA/evidence paths under `.aop/features/**`
+  - legacy-to-`.aop` migration behavior and precedence
+
+### 5.2 Regression Requirements
+
+- Existing CLI contract tests for `run/status/resume/stop` MUST pass unchanged at envelope/exit-code level.
+- Existing MCP transport parity tests MUST pass unchanged for success/failure semantics.
+
+---
+
+## 6. Implementation Milestones
+
+### M19: Path Layout Foundation
+
+- Introduce centralized path abstraction and wire kernel/service/CLI callsites to it.
+
+### M20: Generated Artifact Relocation
+
+- Move runtime-generated write paths to `.aop/features/**` and `.aop/runtime/**`.
+- Keep orchestrator config/schema/tool contracts in `agentic/orchestrator/**`.
+
+### M21: Compatibility + Migration
+
+- Implement deterministic legacy detection and forward migration from `agentic/**` generated roots.
+- Add explicit tests for precedence and no-data-loss behavior.
+
+### M22: Contract/Docs/CI Hardening
+
+- Update README/runtime docs and command examples.
+- Ensure lint/typecheck/test/contract validation gates remain green.
+
+---
+
+## 7. File-Level Target List
+
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/core/schemas.ts` (no root move expected; verify unchanged orchestrator schema root)
+- `apps/control-plane/src/cli/spec-utils.ts`
+- `apps/control-plane/src/cli/spec-input-resolver.ts`
+- `apps/control-plane/src/cli/spec-ingestion-service.ts`
+- `apps/control-plane/src/application/services/feature-lifecycle-service.ts`
+- `apps/control-plane/src/application/services/feature-state-service.ts`
+- `apps/control-plane/src/application/services/plan-service.ts`
+- `apps/control-plane/src/application/services/qa-index-service.ts`
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `apps/control-plane/src/application/services/reporting-service.ts`
+- `apps/control-plane/src/application/services/merge-service.ts`
+- `apps/control-plane/src/mcp/operation-ledger.ts`
+- `apps/control-plane/test/**/*.spec.ts` (path assertions and migration coverage)
+- `README.md`
+- `.gitignore`
+
+---
+
+## 8. Acceptance Criteria
+
+1. New runs create/update runtime artifacts under `.aop/**` only.
+2. No new runtime-generated files are written under `agentic/features/**` or `agentic/runtime/**`.
+3. `feature.discover_specs` and CLI ingestion canonicalize to `.aop/features/<feature_id>/spec.md`.
+4. Legacy repositories with prior `agentic/**` generated artifacts continue to run via deterministic migration/compatibility behavior.
+5. CLI/MCP tool contracts, error envelopes, and deterministic orchestration semantics remain intact.
+6. `npm run lint`, `npm run typecheck`, `npm test`, and `npm run validate:mcp-contracts` pass after migration changes.
+
+---
+
+## 9. Risks and Mitigations
+
+- Risk: Partial path migration causes split state across `agentic/**` and `.aop/**`.
+  - Mitigation: single path authority abstraction + explicit precedence rules + migration tests.
+- Risk: Existing automation/scripts rely on `agentic/features/**`.
+  - Mitigation: compatibility window, README migration notes, and deterministic fallback logic.
+- Risk: Review workflows lose visibility if users are unaware of `.aop/**`.
+  - Mitigation: update README review instructions and CLI output paths.
+
+---
+
+## 10. Dependency Notes
+
+- This spec extends and narrows file-layout behavior from:
+  - `spec-files/agentic_orchestrator_spec.md` (canonical state/runtime contracts)
+  - `spec-files/agentic_orchestrator_mcp_formalization_spec.md` (formal tool/runtime boundary)
+  - `spec-files/agentic_orchestrator_oop_refactor_spec.md` (path abstraction and layered responsibility discipline)
+- Must also updatea progress.md filed `spec-files/progress.md` to reflect this new spec and its milestones as it is completed.

package/spec-files/agentic_orchestrator_mcp_formalization_spec.md

@@ -0,0 +1,379 @@
+# Feature Spec: Formal MCP Tool Contracts and Transport Boundary Refactor (AOP)
+
+> **Purpose of this document**: Define an implementation-ready refactor that converts the current code-first tool surface into formally declared MCP tools with enforced contracts, authenticated role identity, deterministic retries, and transport parity guarantees.
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Capabilities
+
+- Define a single source of truth for each tool:
+  - `name`
+  - `description`
+  - `input_schema`
+  - `output_schema`
+  - supported roles
+  - bound handler
+- Expose a real MCP server adapter that serves:
+  - `tools/list`
+  - `tools/call`
+- Enforce authenticated role identity on every tool call.
+- Preserve deterministic kernel guarantees:
+  - schema validation
+  - optimistic concurrency
+  - lock/collision enforcement
+  - gate/evidence behavior
+  - atomic state writes
+- Enforce idempotency for mutating tool calls using operation keys.
+- Provide transport-agnostic clients for CLI/Supervisor with parity tests.
+- Eliminate documentation drift by generating `tools.md` from tool registry artifacts.
+
+### 1.2 Non-Goals
+
+- No rewrite of core domain behavior in `AopKernel` methods.
+- No change to feature workflow semantics.
+- No provider-specific logic embedded into MCP deterministic handlers.
+
+---
+
+## 2. Current Risks to Resolve
+
+1. Tool contracts are implicit in code and docs, not formal runtime artifacts.
+2. Role authorization is policy-based but caller identity trust model is unspecified for transport calls.
+3. Retry safety is not formally defined at MCP boundary.
+4. RBAC ownership is split (policy and code/docs), enabling drift.
+5. Protocol compatibility is not pinned to an MCP SDK/protocol version.
+
+---
+
+## 3. Architecture Decisions
+
+### 3.1 Tool Registry as Contract Source
+
+All MCP tool metadata MUST be loaded from a registry artifact and not from handwritten switch statements.
+
+### 3.2 Authenticated Identity Model
+
+`actor_type` and `actor_id` are claims, not trusted caller inputs.
+
+- MCP MUST accept a signed session token per tool call.
+- MCP MUST derive actor claims from the token payload.
+- Any supplied actor fields that conflict with token claims MUST fail.
+- `system` role tokens MUST be bootstrap-only and unavailable to worker sessions.
+
+### 3.3 RBAC Precedence Model
+
+RBAC authority is two-level:
+
+1. Tool registry declares `supported_roles` (maximum allowed by implementation).
+2. `policy.yaml.rbac` declares runtime allowlist.
+
+Effective permissions are the intersection. Startup MUST fail if policy grants a role not in `supported_roles`.
+
+### 3.4 Idempotency Model for Mutations
+
+All mutating tools MUST require `operation_id`.
+
+- If an operation with same `operation_id` and same canonical input repeats, MCP returns cached prior result.
+- If `operation_id` is reused with different input, MCP returns `operation_id_reuse_mismatch`.
+- Operation result ledger MUST be durable.
+
+### 3.5 Hybrid Transport Rollout
+
+`ToolClient` abstraction is mandatory:
+- `InProcessToolClient` for tests/fallback.
+- `McpToolClient` for production path.
+
+Production default switches to MCP transport only after parity acceptance gates pass.
+
+---
+
+## 4. Canonical Artifacts
+
+```text
+agentic/
+  orchestrator/
+    tools/
+      catalog.json
+      schemas/
+        input/
+        output/
+      errors.schema.json
+      protocol.json
+  runtime/
+    operation-ledger/
+      <run_id>.json
+```
+
+### 4.1 `catalog.json` Contract
+
+Each tool entry MUST contain:
+- `name`
+- `description`
+- `input_schema_ref`
+- `output_schema_ref`
+- `supported_roles`
+- `handler_id`
+- `mutating` (boolean)
+- `requires_operation_id` (boolean; true when `mutating=true`)
+
+### 4.2 `protocol.json` Contract
+
+Must pin:
+- MCP protocol version supported
+- SDK package + version
+- enabled transport modes (`stdio` required; others optional)
+
+### 4.3 Error Envelope
+
+All failures MUST return:
+- `ok: false`
+- `error.code`
+- `error.message`
+- `error.details`
+- optional `evidence`
+
+---
+
+## 5. Runtime Components
+
+1. `ToolRegistryLoader`
+- loads registry
+- validates uniqueness and schema refs
+
+2. `ToolContractValidator`
+- validates input pre-handler
+- validates output post-handler
+
+3. `TokenAuthVerifier`
+- verifies signature and expiry
+- extracts trusted actor claims
+
+4. `ToolAuthorizer`
+- applies intersection of registry roles and policy roles
+
+5. `OperationLedger`
+- writes and resolves idempotent operation outcomes for mutating calls
+
+6. `KernelToolExecutor`
+- maps `handler_id` to domain method call
+- normalizes errors
+
+7. `McpServerAdapter`
+- implements protocol endpoints via registry/executor
+
+---
+
+## 6. Security and Identity Contract
+
+### 6.1 Token Claims
+
+Required claims:
+- `run_id`
+- `session_id`
+- `actor_type`
+- `actor_id`
+- `issued_at`
+- `expires_at`
+
+Optional claims:
+- `feature_scope` (for feature-limited workers)
+
+### 6.2 Validation Rules
+
+- expired token -> `unauthenticated`
+- invalid signature -> `unauthenticated`
+- actor claim mismatch -> `invalid_actor_claim`
+- unauthorized role/tool -> `forbidden_tool_for_role`
+
+### 6.3 Token Issuance
+
+- Supervisor receives a run bootstrap credential at start.
+- Supervisor issues/requests role-scoped session tokens for orchestrator/planner/builder/qa.
+- Worker tokens MUST NOT carry `system` role.
+
+---
+
+## 7. Idempotency and Retry Contract
+
+### 7.1 Mutating Tools
+
+Mutating tools include at least:
+- `feature.init`
+- `feature.state_patch`
+- `feature.log_append`
+- `plan.submit`
+- `plan.update`
+- `repo.ensure_worktree`
+- `repo.apply_patch`
+- `gates.run`
+- `qa.test_index_update`
+- `locks.acquire`
+- `locks.release`
+- `feature.ready_to_merge`
+
+### 7.2 Required Behavior
+
+- all mutating requests include `operation_id`
+- canonical request hash recorded with result
+- duplicate identical request returns previous response
+- duplicate mismatched request fails with stable error code
+
+---
+
+## 8. Protocol Compatibility and Versioning
+
+### 8.1 Pinning Requirement
+
+Implementation MUST pin and document:
+- MCP protocol version
+- SDK package/version
+- transport implementation details
+
+### 8.2 Compatibility Policy
+
+- breaking contract/schema changes require version bump
+- migration notes required for each breaking change
+- previous minor version accepted during one deprecation window where feasible
+
+---
+
+## 9. Observability Contract
+
+Every tool call MUST emit structured logs with:
+- `run_id`
+- `operation_id` (if mutating)
+- `feature_id` (if applicable)
+- `tool_name`
+- `actor_type`
+- `latency_ms`
+- `result` (`ok|error`)
+- `error_code` (if any)
+
+Metrics MUST include:
+- calls by tool/role
+- failure rates by error code
+- retry and idempotency hit rates
+
+---
+
+## 10. CLI and Supervisor Migration
+
+### 10.1 CLI
+
+- replace direct kernel calls with `ToolClient`
+- add `--transport inprocess|mcp`
+- default transition:
+  - early phases: `inprocess`
+  - post-parity: `mcp`
+
+### 10.2 Supervisor
+
+- replace direct `kernel.invoke(...)` with `ToolClient`
+- preserve role firewall
+- pass signed role token in MCP mode
+
+---
+
+## 11. Integrated Delivery Plan (Cross-Spec)
+
+This plan coordinates this spec and `agentic_orchestrator_single_global_orchestrator_spec.md`.
+
+### M10: Registry Foundation
+
+- implement catalog + schemas + loader + generator
+- bind existing handlers via `handler_id`
+
+Exit gate:
+- registry validates
+- generated tools doc matches registry
+
+### M11: Auth + Idempotency Core
+
+- token verifier and role-claim enforcement
+- operation ledger and mutating call idempotency
+
+Exit gate:
+- auth failure paths tested
+- duplicate mutation calls are safe
+
+### M12: MCP Adapter + Protocol Pin
+
+- implement MCP adapter (`tools/list`, `tools/call`)
+- pin protocol and sdk versions
+
+Exit gate:
+- protocol conformance tests pass
+
+### M13: ToolClient Migration
+
+- add in-process and MCP clients
+- migrate CLI/Supervisor call paths
+
+Exit gate:
+- end-to-end runs on both transports
+
+### M14-M17: Global Orchestrator Topology Refactor
+
+- executed per global-orchestrator spec milestones
+
+Exit gate:
+- session topology invariants pass (`1 + 3N`)
+
+### M18: Production Promotion
+
+- default transport `mcp`
+- maintain in-process fallback for tests/recovery only
+
+Exit gate:
+- parity suite green
+- rollout checklist signed off
+
+---
+
+## 12. Testing and Verification
+
+### 12.1 Contract Tests
+
+- all tools have valid input/output schema
+- no duplicate names/handler ids
+- `tools/list` equals registry contents
+
+### 12.2 Conformance Suite
+
+Same scenario must pass via both transports for critical paths:
+- init -> plan -> patch -> gates -> merge
+- lock conflict and collision outcomes
+- retried mutation calls
+- crash/recovery flows
+
+### 12.3 Security Tests
+
+- token expiry/signature rejection
+- role spoofing rejection
+- system-role misuse rejection
+
+---
+
+## 13. Acceptance Criteria
+
+1. Tool contracts are runtime-loaded from registry artifacts.
+2. Every tool call is authenticated and role-derived from verified token claims.
+3. Mutating tool calls are idempotent by `operation_id`.
+4. MCP transport behavior is parity-verified against in-process transport.
+5. RBAC enforcement follows defined intersection model without drift.
+6. Protocol version and SDK version are pinned and validated in CI.
+7. `tools.md` is generated from registry and no longer manually authored.
+
+---
+
+## 14. Risks and Mitigations
+
+- Risk: added complexity from auth and idempotency layers.
+  - Mitigation: isolate into dedicated components and exhaustively unit test.
+- Risk: rollout instability from transport migration.
+  - Mitigation: keep in-process fallback until M18 exit gate.
+- Risk: schema churn causing breakages.
+  - Mitigation: strict versioning and migration policy.
+