agentic-orchestrator 0.1.2 → 0.1.4

package/spec-files/outstanding/agentic_orchestrator_enterprise_governance_dashboard_spec.md

@@ -0,0 +1,348 @@
+# Feature Spec: Enterprise Governance and Secure Dashboard Operations (AOP)
+
+> **Purpose of this document**: Define implementation-ready delivery for Q5 and Q6: compliance policy packs + control export, and production-grade dashboard authentication/authorization with audit federation.
+
+**Version:** 1.0  
+**Date:** 2026-03-03  
+**Status:** Draft  
+**Roadmap Mapping:** M37
+
+---
+
+## 0. Scope and Standards
+
+### 0.1 Feature Scope
+
+This spec implements:
+- **Q5** Compliance Policy Packs + Control Export
+- **Q6** Dashboard AuthN/AuthZ + SSO + Audit Federation
+
+### 0.2 Required Standards
+
+Implementation MUST preserve:
+- deterministic evidence mapping
+- explicit authorization for all mutating dashboard actions
+- audit-log completeness for privileged operations
+- optional/off-by-default behavior for local developer environments
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- operators can apply standardized compliance packs and export machine-readable control evidence mappings
+- dashboard can run safely in multi-user environments with OIDC/API key auth and RBAC
+- privileged dashboard actions emit structured audit records suitable for SIEM ingestion
+
+### 1.2 Non-Goals
+
+- no claim of automatic certification/compliance attestation
+- no external IAM provisioning automation in this phase
+- no hard requirement for auth in local dev mode
+
+---
+
+## 2. Architecture Decisions
+
+### 2.1 Compliance as Policy Overlay
+
+Compliance packs are deterministic overlays evaluated against existing policy, runtime state, and evidence artifacts. They do not replace base policy configuration.
+
+### 2.2 Dashboard Security Boundary
+
+Dashboard API authorization is enforced server-side for every action endpoint; UI affordances alone are insufficient and non-authoritative.
+
+### 2.3 Unified Audit Envelope
+
+Audit events from dashboard actions and compliance exports share a common schema for downstream ingestion.
+
+---
+
+## 3. Contracts and Config
+
+### 3.1 Policy Additions
+
+```yaml
+innovation:
+  compliance_packs: false
+  dashboard_auth: false
+
+compliance:
+  enabled: false
+  default_pack: soc2_baseline
+  export:
+    include_manual_controls: true
+
+dashboard:
+  auth:
+    enabled: false
+    mode: oidc_or_api_key  # oidc | api_key | oidc_or_api_key
+    oidc:
+      issuer: ""
+      client_id: ""
+      audience: ""
+    api_keys:
+      env_var: AOP_DASHBOARD_API_KEYS
+  rbac:
+    viewer: [status_read, feature_read, evidence_read]
+    reviewer: [status_read, feature_read, evidence_read, review_decide]
+    operator: [status_read, feature_read, evidence_read, review_decide, feature_checkout]
+    admin: ["*"]
+  audit:
+    enabled: true
+    sink: file  # file | webhook
+    webhook_url: ""
+```
+
+### 3.2 Source-Managed Compliance Assets
+
+```text
+agentic/orchestrator/compliance/packs/
+  soc2_baseline.yaml
+  iso27001_baseline.yaml
+  hipaa_baseline.yaml
+agentic/orchestrator/schemas/compliance_pack.schema.json
+```
+
+### 3.3 Runtime Artifacts
+
+```text
+.aop/runtime/compliance/control_export_<timestamp>.json
+.aop/runtime/audit/dashboard-audit.jsonl
+```
+
+### 3.4 New MCP Tool
+
+- `compliance.export_controls`
+
+### 3.5 New CLI Command
+
+- `aop compliance export --format <json|csv> [--pack <id>] [--include-manual true|false]`
+
+---
+
+## 4. Detailed Feature Specs
+
+## 4.1 Q5: Compliance Policy Packs + Control Export
+
+### 4.1.1 Problem
+Enterprise adoption requires consistent control mapping from orchestrator behavior/evidence to recognized control frameworks.
+
+### 4.1.2 Design
+Provide curated compliance packs plus deterministic export routine.
+
+### 4.1.3 Pack Structure
+
+```yaml
+id: soc2_baseline
+version: "1.0"
+controls:
+  - id: CC7.2
+    title: Monitor system components for anomalies
+    requirements:
+      - path: policy.testing.coverage.minimums.line
+        op: gte
+        value: 0.9
+      - path: policy.merge_policy.require_user_approval
+        op: eq
+        value: true
+    evidence_sources:
+      - gate_results
+      - operation_ledger
+    manual_evidence_required: false
+```
+
+### 4.1.4 Export Output Contract
+
+```json
+{
+  "pack_id": "soc2_baseline",
+  "generated_at": "2026-03-03T12:00:00Z",
+  "controls": [
+    {
+      "id": "CC7.2",
+      "status": "pass",
+      "automated_checks": [
+        {
+          "path": "policy.testing.coverage.minimums.line",
+          "result": "pass",
+          "observed": 0.9
+        }
+      ],
+      "evidence_refs": [
+        ".aop/features/feature_x/evidence/gates/full-2026-03-03.json"
+      ],
+      "manual_evidence_required": false
+    }
+  ]
+}
+```
+
+### 4.1.5 File Targets
+
+- `apps/control-plane/src/application/services/compliance-service.ts` (new)
+- `apps/control-plane/src/cli/compliance-export-command-handler.ts` (new)
+- `agentic/orchestrator/compliance/packs/*.yaml` (new)
+- `agentic/orchestrator/schemas/compliance_pack.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/input/compliance.export_controls.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/compliance.export_controls.output.schema.json` (new)
+
+### 4.1.6 Acceptance Criteria
+
+- compliance packs schema-validate
+- export output is deterministic for same repo state
+- controls include explicit pass/fail and evidence references
+
+---
+
+## 4.2 Q6: Dashboard AuthN/AuthZ + SSO + Audit Federation
+
+### 4.2.1 Problem
+Dashboard operation in shared/production contexts needs strong identity + authorization + audit controls.
+
+### 4.2.2 Design
+Add optional auth middleware, server-side action authorization, and audit sink integration.
+
+### 4.2.3 Auth Modes
+
+- `oidc`: bearer token validation against OIDC issuer metadata
+- `api_key`: static/scoped keys from environment
+- `oidc_or_api_key`: allow either for migration convenience
+
+### 4.2.4 Action Authorization Matrix
+
+Actions to guard:
+- `feature_review_decide` (approve/deny/request_changes)
+- `feature_checkout`
+- `dashboard_admin_settings`
+
+Each action must map to role permission in policy.
+
+### 4.2.5 Audit Event Contract
+
+```json
+{
+  "ts": "2026-03-03T12:10:00Z",
+  "actor": {
+    "id": "alice@example.com",
+    "auth_type": "oidc",
+    "role": "reviewer"
+  },
+  "action": "feature_review_decide",
+  "feature_id": "payment_retry_guard",
+  "result": "allow",
+  "request_id": "req-123",
+  "metadata": {
+    "decision": "approve"
+  }
+}
+```
+
+### 4.2.6 File Targets
+
+- `packages/web-dashboard/src/middleware.ts` (new)
+- `packages/web-dashboard/src/lib/authz.ts` (new)
+- `packages/web-dashboard/src/app/api/auth/*` (new)
+- `packages/web-dashboard/src/app/api/features/[id]/review/route.ts`
+- `packages/web-dashboard/src/app/api/features/[id]/checkout/route.ts`
+- `apps/control-plane/src/application/services/dashboard-audit-service.ts` (new)
+
+### 4.2.7 Acceptance Criteria
+
+- unauthorized users cannot invoke protected dashboard actions
+- audit events emitted for all protected action attempts (allow + deny)
+- local mode works with auth disabled by default
+
+---
+
+## 5. Milestones
+
+### EGD1: Compliance Pack Foundation
+
+Tasks:
+1. Add compliance pack assets + schemas.
+2. Implement export service + MCP tool + CLI command.
+
+Exit gate:
+- deterministic export and evidence mapping verified by tests.
+
+### EGD2: Dashboard Auth and RBAC
+
+Tasks:
+1. Add auth middleware and token/api key validation.
+2. Implement server-side role authorization for protected endpoints.
+
+Exit gate:
+- protected actions denied when unauthorized in integration tests.
+
+### EGD3: Audit Federation
+
+Tasks:
+1. Add unified audit envelope and file sink.
+2. Add optional webhook sink with retry/backoff.
+
+Exit gate:
+- allow/deny events emitted and schema-valid in both sinks.
+
+---
+
+## 6. Testing Strategy (Normative)
+
+### 6.1 Unit Tests
+
+- compliance rule evaluation operators (`eq`, `gte`, etc.)
+- pack schema validation and export assembly
+- auth token/api key parsing and role mapping
+- audit event envelope validation
+
+### 6.2 Integration Tests
+
+- `aop compliance export` on sample repository state
+- dashboard protected endpoint authorization matrix
+- audit logs emitted for protected action attempts
+
+### 6.3 Security Tests
+
+- invalid/expired oidc tokens rejected
+- malformed or unknown API keys rejected
+- privilege escalation attempt denied (`viewer` invoking `review_decide`)
+
+### 6.4 Contract and Parity Tests
+
+- `compliance.export_controls` schema and transport parity (in-process vs MCP)
+
+---
+
+## 7. Acceptance Criteria
+
+1. Compliance packs and export path are fully operational and deterministic.
+2. Dashboard auth/authz can be enabled for production multi-user use.
+3. Audit federation records all protected dashboard action attempts.
+4. Feature flags/defaults preserve existing local development behavior.
+5. Full validation gates pass (`lint`, `typecheck`, `test`, contract/architecture validation).
+
+---
+
+## 8. Risks and Mitigations
+
+- Risk: users assume compliance export equals certification.
+  - Mitigation: mandatory disclaimer and `manual_evidence_required` field.
+
+- Risk: auth misconfiguration locks out operators.
+  - Mitigation: startup preflight checks and explicit break-glass local mode.
+
+- Risk: webhook audit sink failures lose records.
+  - Mitigation: durable local fallback log + retry policy.
+
+---
+
+## 9. Definition of Done
+
+A completed EGD milestone MUST include:
+- changed files and architecture rationale
+- schema/catalog/policy diffs
+- security + integration test coverage
+- gate command results
+- updates in `spec-files/progress.md`
+

package/spec-files/outstanding/agentic_orchestrator_knowledge_canary_spec.md

@@ -0,0 +1,344 @@
+# Feature Spec: Cross-Feature Knowledge Graph and Progressive Merge Canary (AOP)
+
+> **Purpose of this document**: Define implementation-ready delivery for Q10 and Q12: reusable cross-feature knowledge retrieval and deterministic post-merge canary verification guardrails.
+
+**Version:** 1.0  
+**Date:** 2026-03-03  
+**Status:** Draft  
+**Roadmap Mapping:** M38
+
+---
+
+## 0. Scope and Constraints
+
+### 0.1 Feature Scope
+
+This spec implements:
+- **Q10** Cross-Feature Knowledge Graph
+- **Q12** Progressive Merge Guardrails (Canary Verification)
+
+### 0.2 Constraints
+
+- knowledge retrieval MUST be deterministic and auditable
+- canary verification MUST never bypass explicit merge approval requirements
+- canary failure MUST not auto-revert by default
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- planner/orchestrator can retrieve prior feature patterns (collisions, mitigations, gate failures) to improve planning quality
+- merge promotion can optionally run post-merge canary checks before final merged-state confirmation
+- both features remain opt-in and backward compatible
+
+### 1.2 Non-Goals
+
+- no vector DB dependency in this phase
+- no AI ranking/reranking in first iteration
+- no automatic rollback orchestration
+
+---
+
+## 2. Architecture Decisions
+
+### 2.1 Graph as Derived Runtime Artifact
+
+Knowledge graph is deterministic derived state from canonical runtime artifacts (`state.md`, `plan.json`, decisions, gate evidence, collisions, locks). It is not manually edited.
+
+### 2.2 Controlled Retrieval API
+
+Knowledge access is through explicit tool/query APIs with bounded filters and deterministic ranking rules.
+
+### 2.3 Canary as Merge Sub-Stage
+
+Canary verification is implemented as an optional sub-stage in `feature.ready_to_merge` flow. Standard preconditions still execute first.
+
+---
+
+## 3. Contracts and Artifacts
+
+### 3.1 Knowledge Graph Artifact
+
+```text
+.aop/knowledge/graph.json
+```
+
+Node contract:
+
+```json
+{
+  "id": "node:feature:payment_retry_guard",
+  "type": "feature",
+  "labels": ["payments", "retry"],
+  "attrs": {
+    "status": "merged",
+    "created_at": "2026-03-01T11:00:00Z"
+  }
+}
+```
+
+Edge contract:
+
+```json
+{
+  "id": "edge:blocked_by:feature_a:feature_b",
+  "type": "blocked_by",
+  "from": "node:feature:feature_a",
+  "to": "node:feature:feature_b",
+  "attrs": {
+    "reason": "openapi_operation_conflict"
+  }
+}
+```
+
+### 3.2 Canary State Additions (Optional)
+
+`state.md` frontmatter optional fields:
+- `canary.status`: `na|pending|pass|fail`
+- `canary.last_run_at`: RFC3339
+- `canary.evidence_refs`: string[]
+
+### 3.3 New MCP Tools
+
+- `knowledge.search`
+- `merge.canary_verify`
+
+---
+
+## 4. Detailed Feature Specs
+
+## 4.1 Q10: Cross-Feature Knowledge Graph
+
+### 4.1.1 Problem
+Historical delivery knowledge exists but is difficult to query/reuse during planning.
+
+### 4.1.2 Design
+Build and maintain graph from canonical artifacts and expose deterministic search API.
+
+### 4.1.3 Node Types
+
+- `feature`
+- `file`
+- `lock_resource`
+- `collision`
+- `gate_failure`
+- `mitigation_pattern`
+
+### 4.1.4 Edge Types
+
+- `touches`
+- `blocked_by`
+- `fixed_by`
+- `depends_on`
+- `reuses_pattern`
+
+### 4.1.5 Ranking Rules (Deterministic)
+
+Ranking by tuple order:
+1. exact feature/status/type match
+2. recency descending
+3. frequency of relevant edge types
+4. lexicographic node id
+
+### 4.1.6 Query Contract
+
+Input (`knowledge.search`):
+
+```json
+{
+  "query": "db migration retry lock",
+  "feature_id": "payment_retry_guard",
+  "limit": 20,
+  "filters": {
+    "node_types": ["mitigation_pattern", "gate_failure"]
+  }
+}
+```
+
+Output:
+
+```json
+{
+  "results": [
+    {
+      "node_id": "node:mitigation:retry_lock_backoff",
+      "score": 0.89,
+      "evidence_refs": [
+        ".aop/features/legacy_feature/evidence/gates/full-2026-02-20.json"
+      ]
+    }
+  ]
+}
+```
+
+### 4.1.7 File Targets
+
+- `apps/control-plane/src/application/services/knowledge-graph-service.ts` (new)
+- `apps/control-plane/src/application/services/plan-service.ts`
+- `apps/control-plane/src/application/services/feature-state-service.ts`
+- `agentic/orchestrator/tools/schemas/input/knowledge.search.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/knowledge.search.output.schema.json` (new)
+
+### 4.1.8 Acceptance Criteria
+
+- graph generated/updated deterministically from canonical sources
+- search returns stable ordering for same query and graph state
+- planner can consume top results as optional context bundle extension
+
+---
+
+## 4.2 Q12: Progressive Merge Guardrails (Canary Verification)
+
+### 4.2.1 Problem
+Pre-merge checks can miss issues that only surface on merged head.
+
+### 4.2.2 Design
+Optional post-merge canary gate profile runs before final merged-state confirmation.
+
+### 4.2.3 Canary Flow
+
+1. `feature.ready_to_merge` validates existing preconditions (approval, required gates, status).
+2. perform merge operation in deterministic path.
+3. if `innovation.canary_merge_verification=true`, run canary profile.
+4. if canary passes:
+   - set `canary.status=pass`
+   - finalize `merged`
+5. if canary fails:
+   - set `canary.status=fail`
+   - set feature `status=blocked`
+   - emit rollback guidance artifact with deterministic remediation steps.
+
+### 4.2.4 Policy Additions
+
+```yaml
+innovation:
+  canary_merge_verification: false
+
+merge_policy:
+  canary:
+    profile: canary
+    mode: post_merge
+    fail_behavior: block_with_guidance
+```
+
+### 4.2.5 Rollback Guidance Artifact
+
+```text
+.aop/features/<feature_id>/evidence/canary_rollback_guidance_<timestamp>.md
+```
+
+Includes:
+- failing canary steps
+- relevant logs/evidence refs
+- deterministic git commands to revert/fix-forward (suggested, not auto-executed)
+
+### 4.2.6 File Targets
+
+- `apps/control-plane/src/application/services/canary-verification-service.ts` (new)
+- `apps/control-plane/src/application/services/merge-service.ts`
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/schemas/state.schema.json`
+- `agentic/orchestrator/tools/schemas/input/merge.canary_verify.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/merge.canary_verify.output.schema.json` (new)
+
+### 4.2.7 Acceptance Criteria
+
+- canary path opt-in and disabled by default
+- canary pass/fail state persisted deterministically
+- fail path blocks final merged confirmation and emits guidance artifact
+
+---
+
+## 5. Milestones
+
+### KC1: Knowledge Graph Foundation
+
+Tasks:
+1. implement graph model/service and build/update pipeline.
+2. add search tool contracts and query API.
+
+Exit gate:
+- graph file and search output deterministic under repeated runs.
+
+### KC2: Planner Context Integration
+
+Tasks:
+1. wire optional knowledge retrieval into planner context bundle.
+2. add policy toggle for knowledge enrichment usage.
+
+Exit gate:
+- planner receives contextual patterns without changing core deterministic flow.
+
+### KC3: Canary Merge Verification
+
+Tasks:
+1. implement canary verification service.
+2. integrate into merge-service finalization path.
+3. emit rollback guidance on fail.
+
+Exit gate:
+- merge path with canary enabled handles pass/fail transitions correctly.
+
+---
+
+## 6. Testing Strategy (Normative)
+
+### 6.1 Unit Tests
+
+- graph node/edge dedup and deterministic sort order
+- knowledge ranking stability for fixed input
+- canary pass/fail transition logic
+- rollback guidance content completeness
+
+### 6.2 Integration Tests
+
+- graph updates after feature lifecycle events
+- `knowledge.search` returns expected patterns for seeded artifacts
+- `feature.ready_to_merge` with canary enabled:
+  - success path to `merged`
+  - failure path to `blocked` + guidance artifact
+
+### 6.3 Transport Parity
+
+- `knowledge.search` parity (in-process vs MCP)
+- `merge.canary_verify` parity (in-process vs MCP)
+
+---
+
+## 7. Acceptance Criteria
+
+1. Knowledge graph built and queryable with deterministic ranking.
+2. Planner can consume knowledge results in controlled opt-in mode.
+3. Canary verification can protect merge finalization without breaking existing default behavior.
+4. New tools, schemas, and policies validate and pass all quality gates.
+
+---
+
+## 8. Risks and Mitigations
+
+- Risk: graph growth impacts performance.
+  - Mitigation: bounded compaction and deterministic archival windows.
+
+- Risk: noisy knowledge retrieval overwhelms planner context.
+  - Mitigation: strict limit/filter controls and deterministic top-K selection.
+
+- Risk: canary stage increases merge latency.
+  - Mitigation: lightweight canary profile and policy tuning.
+
+- Risk: teams misinterpret blocked-after-canary as merge corruption.
+  - Mitigation: explicit status reason and rollback/fix-forward guidance artifact.
+
+---
+
+## 9. Definition of Done
+
+A completed KC milestone MUST include:
+- changed files and rationale
+- policy/schema/tool catalog diffs
+- integration + parity tests
+- command verification summary
+- `spec-files/progress.md` update with residual tasks (if any)
+