agentic-orchestrator 0.1.28 → 0.2.0

package/spec-files/outstanding/agentic_orchestrator_knowledge_canary_spec.md

@@ -1,28 +1,80 @@
-# Feature Spec: Cross-Feature Knowledge Graph and Progressive Merge Canary (AOP)
+# Feature Spec: Cross-Feature Knowledge Graph (AOP)
 
-> **Purpose of this document**: Define implementation-ready delivery for Q10 and Q12: reusable cross-feature knowledge retrieval and deterministic post-merge canary verification guardrails.
+> **Purpose of this document**: Define implementation-ready delivery for Q10 and record the architectural decision to defer the original Q12 post-merge canary design now that the shared merge reconciler exists.
 
-**Version:** 1.0  
-**Date:** 2026-03-03  
-**Status:** Draft  
+**Version:** 2.0  
+**Date:** 2026-03-16  
+**Status:** Draft (Q10 Active, Q12 Deferred)  
 **Roadmap Mapping:** M38
 
 ---
 
-## 0. Scope and Constraints
+## 0. Scope and Decision Record
 
-### 0.1 Feature Scope
+### 0.1 In-Scope
 
 This spec implements:
 
 - **Q10** Cross-Feature Knowledge Graph
-- **Q12** Progressive Merge Guardrails (Canary Verification)
 
-### 0.2 Constraints
+### 0.2 Deferred From This Spec
+
+This spec does **not** implement the original Q12 "post-merge canary verification" flow.
+
+Decision:
+
+- the current shared reconciler already addresses the primary risk class that the old canary design was trying to mitigate: mainline divergence that only becomes visible when reconciling a feature against updated mainline state.
+- the current merge path already requires required gate modes to pass before merge promotion.
+- the old Q12 design added a heavyweight post-merge blocked-state lifecycle that now overlaps with the reconciler's purpose while also conflicting with current merge cleanup semantics.
+
+Therefore:
+
+- **Q12 is deferred** from M38 in its original form.
+- if the project later proves a need for true merged-head-only verification, that should be specified as a separate, smaller feature focused on merged-head confirmation before cleanup, not as the original blocked-after-merge canary workflow.
+
+### 0.3 Constraints
 
 - knowledge retrieval MUST be deterministic and auditable
-- canary verification MUST never bypass explicit merge approval requirements
-- canary failure MUST not auto-revert by default
+- no vector DB dependency in this phase
+- no AI ranking/reranking in first iteration
+- merged feature cleanup MUST NOT destroy the historical facts needed for deterministic retrieval
+
+### 0.4 Current Project State
+
+This spec was audited against the live repository on 2026-03-16.
+
+Current status:
+
+1. **Q10 is still unimplemented.**
+   - there is no `knowledge.search` tool, no knowledge graph service, and no persisted knowledge graph artifact today.
+
+2. **The reconciler materially changes the old Q12 tradeoff.**
+   - mainline divergence is already routed into the shared reconciler lane.
+   - reconciler guidance already requires verification via `gates.run` fast mode, with escalation to fuller verification when needed.
+   - this leaves only a narrow "true merged-head-only" risk class for any future Q12 replacement.
+
+3. **Current merge cleanup matters for Q10.**
+   - `MergeService.featureReadyToMerge()` can immediately delete `.aop/features/<feature_id>` when `cleanup.auto_after_merge=true` and `grace_period_seconds<=0`.
+   - Q10 therefore cannot assume merged feature artifacts remain on disk indefinitely.
+   - implementation MUST define a deterministic retention/snapshot rule before enabling Q10.
+
+4. **Replay exists and may be consumed, but evidence-integrity and doctor remain optional future inputs.**
+   - replay timeline is implemented and is available as an auditable historical source.
+   - evidence-integrity and doctor work remain outstanding and MUST NOT be required for Q10 v1.
+
+### 0.5 Current Architecture Wiring Requirements
+
+Implementation must update the current contract surfaces, not older pre-refactor seams:
+
+- `agentic/orchestrator/defaults/policy.defaults.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/schemas/policy.user.schema.json`
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/application/tools/tool-metadata.ts`
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/application/kernel-tool-wiring.ts`
+- `agentic/orchestrator/tools/catalog.json`
+- regenerated `agentic/orchestrator/tools.md`
 
 ---
 
@@ -30,15 +82,15 @@ This spec implements:
 
 ### 1.1 Must-Have Outcomes
 
-- planner/orchestrator can retrieve prior feature patterns (collisions, mitigations, gate failures) to improve planning quality
-- merge promotion can optionally run post-merge canary checks before final merged-state confirmation
-- both features remain opt-in and backward compatible
+- planner/orchestrator can retrieve prior feature patterns such as collisions, mitigations, gate failures, and dependency history to improve plan quality
+- historical retrieval remains deterministic, bounded, and auditable
+- the feature remains opt-in and backward compatible
 
 ### 1.2 Non-Goals
 
-- no vector DB dependency in this phase
-- no AI ranking/reranking in first iteration
-- no automatic rollback orchestration
+- no live semantic search or embedding pipeline
+- no autonomous planner prompt rewriting
+- no post-merge canary state machine in this milestone
 
 ---
 
@@ -46,15 +98,47 @@ This spec implements:
 
 ### 2.1 Graph as Derived Runtime Artifact
 
-Knowledge graph is deterministic derived state from canonical runtime artifacts (`state.md`, `plan.json`, decisions, gate evidence, collisions, locks). It is not manually edited.
+Knowledge graph is deterministic derived state from canonical runtime artifacts such as:
+
+- `state.md`
+- `plan.json`
+- decisions and logs
+- gate evidence
+- collisions
+- dependency-blocking data
+- replay timeline entries
+- merge-time summaries persisted for retention
+
+It is not manually edited.
+
+### 2.2 Retention Before Cleanup
+
+Because merged feature directories may be auto-cleaned, the graph builder MUST NOT rely on long-term presence of `.aop/features/<id>` after merge.
+
+Implementation MUST choose one of these deterministic patterns:
+
+1. persist graph facts directly into `.aop/knowledge/graph.json` before cleanup can remove the source feature directory, or
+2. persist a smaller merge-time rollup artifact under `.aop/runtime/**` or `.aop/knowledge/**` and build the graph from those retained summaries plus live non-terminal feature artifacts.
+
+Whichever pattern is chosen, the retained inputs must be:
 
-### 2.2 Controlled Retrieval API
+- schema-backed
+- append-safe or lock-safe
+- reconstructable deterministically from canonical runtime state at the time they were captured
 
-Knowledge access is through explicit tool/query APIs with bounded filters and deterministic ranking rules.
+### 2.3 Controlled Retrieval API
 
-### 2.3 Canary as Merge Sub-Stage
+Knowledge access is through explicit tool/query APIs with bounded filters and deterministic ranking rules. Search results must be explainable from persisted graph content, not opaque scoring heuristics.
 
-Canary verification is implemented as an optional sub-stage in `feature.ready_to_merge` flow. Standard preconditions still execute first.
+### 2.4 Q12 Replacement Guidance
+
+The original canary concept is intentionally not carried forward here.
+
+If a future follow-up is required, it should target:
+
+- merged-head-only regressions that neither pre-merge gates nor the reconciler already cover
+- a small verification step before cleanup/final confirmation
+- no new long-lived `blocked-after-merge` lifecycle unless a concrete operational need is proven
 
 ---
 
@@ -94,32 +178,46 @@ Edge contract:
 }
 ```
 
-### 3.2 Canary State Additions (Optional)
+### 3.2 New MCP Tool
 
-`state.md` frontmatter optional fields:
+- `knowledge.search`
 
-- `canary.status`: `na|pending|pass|fail`
-- `canary.last_run_at`: RFC3339
-- `canary.evidence_refs`: string[]
+No `merge.canary_verify` tool is defined by this revision.
 
-### 3.3 New MCP Tools
+### 3.3 Optional Policy Additions
 
-- `knowledge.search`
-- `merge.canary_verify`
+```yaml
+innovation:
+  knowledge_graph: false
+
+knowledge:
+  enabled: false
+  planner_context:
+    enabled: false
+    max_results: 5
+  retention:
+    mode: graph_snapshot # graph_snapshot | merge_rollup
+```
+
+Rules:
+
+- all toggles default to `false`
+- `innovation.knowledge_graph` and `knowledge.enabled` must both be `true` before graph build/search is active
+- `planner_context.enabled` only affects whether planner context is enriched; it does not change graph generation semantics
 
 ---
 
-## 4. Detailed Feature Specs
+## 4. Detailed Feature Spec
 
 ## 4.1 Q10: Cross-Feature Knowledge Graph
 
 ### 4.1.1 Problem
 
-Historical delivery knowledge exists but is difficult to query/reuse during planning.
+Historical delivery knowledge exists but is difficult to query and reuse during planning, especially once merged feature directories have been cleaned up.
 
 ### 4.1.2 Design
 
-Build and maintain graph from canonical artifacts and expose deterministic search API.
+Build and maintain a deterministic graph from canonical runtime artifacts and retained merge-time summaries, then expose a deterministic search API.
 
 ### 4.1.3 Node Types
 
@@ -129,6 +227,7 @@ Build and maintain graph from canonical artifacts and expose deterministic searc
 - `collision`
 - `gate_failure`
 - `mitigation_pattern`
+- `dependency`
 
 ### 4.1.4 Edge Types
 
@@ -137,6 +236,7 @@ Build and maintain graph from canonical artifacts and expose deterministic searc
 - `fixed_by`
 - `depends_on`
 - `reuses_pattern`
+- `failed_gate_on`
 
 ### 4.1.5 Ranking Rules (Deterministic)
 
@@ -180,119 +280,51 @@ Output:
 
 - `apps/control-plane/src/application/services/knowledge-graph-service.ts` (new)
 - `apps/control-plane/src/application/services/plan-service.ts`
-- `apps/control-plane/src/application/services/feature-state-service.ts`
+- `apps/control-plane/src/supervisor/planning-wave-executor.ts`
+- `apps/control-plane/src/application/services/merge-service.ts`
+- `apps/control-plane/src/core/path-layout.ts`
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/application/tools/tool-metadata.ts`
+- `agentic/orchestrator/defaults/policy.defaults.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/schemas/policy.user.schema.json`
+- `agentic/orchestrator/tools/catalog.json`
 - `agentic/orchestrator/tools/schemas/input/knowledge.search.input.schema.json` (new)
 - `agentic/orchestrator/tools/schemas/output/knowledge.search.output.schema.json` (new)
 
 ### 4.1.8 Acceptance Criteria
 
-- graph generated/updated deterministically from canonical sources
-- search returns stable ordering for same query and graph state
-- planner can consume top results as optional context bundle extension
-
----
-
-## 4.2 Q12: Progressive Merge Guardrails (Canary Verification)
-
-### 4.2.1 Problem
-
-Pre-merge checks can miss issues that only surface on merged head.
-
-### 4.2.2 Design
-
-Optional post-merge canary gate profile runs before final merged-state confirmation.
-
-### 4.2.3 Canary Flow
-
-1. `feature.ready_to_merge` validates existing preconditions (approval, required gates, status).
-2. perform merge operation in deterministic path.
-3. if `innovation.canary_merge_verification=true`, run canary profile.
-4. if canary passes:
-   - set `canary.status=pass`
-   - finalize `merged`
-5. if canary fails:
-   - set `canary.status=fail`
-   - set feature `status=blocked`
-   - emit rollback guidance artifact with deterministic remediation steps.
-
-### 4.2.4 Policy Additions
-
-```yaml
-innovation:
-  canary_merge_verification: false
-
-merge_policy:
-  canary:
-    profile: canary
-    mode: post_merge
-    fail_behavior: block_with_guidance
-```
-
-### 4.2.5 Rollback Guidance Artifact
-
-```text
-.aop/features/<feature_id>/evidence/canary_rollback_guidance_<timestamp>.md
-```
-
-Includes:
-
-- failing canary steps
-- relevant logs/evidence refs
-- deterministic git commands to revert/fix-forward (suggested, not auto-executed)
-
-### 4.2.6 File Targets
-
-- `apps/control-plane/src/application/services/canary-verification-service.ts` (new)
-- `apps/control-plane/src/application/services/merge-service.ts`
-- `apps/control-plane/src/application/services/gate-service.ts`
-- `agentic/orchestrator/schemas/policy.schema.json`
-- `agentic/orchestrator/schemas/state.schema.json`
-- `agentic/orchestrator/tools/schemas/input/merge.canary_verify.input.schema.json` (new)
-- `agentic/orchestrator/tools/schemas/output/merge.canary_verify.output.schema.json` (new)
-
-### 4.2.7 Acceptance Criteria
-
-- canary path opt-in and disabled by default
-- canary pass/fail state persisted deterministically
-- fail path blocks final merged confirmation and emits guidance artifact
+- graph generated or updated deterministically from canonical sources plus the selected retention mechanism
+- search returns stable ordering for the same query and graph state
+- planner can consume top results as an optional context bundle extension
+- merged-feature cleanup does not destroy the historical facts required for future deterministic search
 
 ---
 
 ## 5. Milestones
 
-### KC1: Knowledge Graph Foundation
-
-Tasks:
-
-1. implement graph model/service and build/update pipeline.
-2. add search tool contracts and query API.
-
-Exit gate:
-
-- graph file and search output deterministic under repeated runs.
-
-### KC2: Planner Context Integration
+### KG1: Graph Foundation
 
 Tasks:
 
-1. wire optional knowledge retrieval into planner context bundle.
-2. add policy toggle for knowledge enrichment usage.
+1. implement graph model/service and build/update pipeline
+2. define and implement the retention boundary so graph facts survive current auto-cleanup behavior
+3. add search tool contracts and query API
 
 Exit gate:
 
-- planner receives contextual patterns without changing core deterministic flow.
+- graph file and search output are deterministic under repeated runs
 
-### KC3: Canary Merge Verification
+### KG2: Planner Context Integration
 
 Tasks:
 
-1. implement canary verification service.
-2. integrate into merge-service finalization path.
-3. emit rollback guidance on fail.
+1. wire optional knowledge retrieval into the planner context bundle
+2. add policy toggles for knowledge graph generation and planner enrichment usage
 
 Exit gate:
 
-- merge path with canary enabled handles pass/fail transitions correctly.
+- planner receives contextual patterns without changing core deterministic flow when the feature is disabled
 
 ---
 
@@ -302,30 +334,27 @@ Exit gate:
 
 - graph node/edge dedup and deterministic sort order
 - knowledge ranking stability for fixed input
-- canary pass/fail transition logic
-- rollback guidance content completeness
+- retention/snapshot path preserves graph facts across cleanup
 
 ### 6.2 Integration Tests
 
 - graph updates after feature lifecycle events
 - `knowledge.search` returns expected patterns for seeded artifacts
-- `feature.ready_to_merge` with canary enabled:
-  - success path to `merged`
-  - failure path to `blocked` + guidance artifact
+- merged feature cleanup still leaves graph/search data intact
 
 ### 6.3 Transport Parity
 
 - `knowledge.search` parity (in-process vs MCP)
-- `merge.canary_verify` parity (in-process vs MCP)
 
 ---
 
 ## 7. Acceptance Criteria
 
-1. Knowledge graph built and queryable with deterministic ranking.
+1. Knowledge graph is built and queryable with deterministic ranking.
 2. Planner can consume knowledge results in controlled opt-in mode.
-3. Canary verification can protect merge finalization without breaking existing default behavior.
+3. Merge cleanup, feature deletion, and retained-history behavior remain deterministic under the selected retention strategy.
 4. New tools, schemas, and policies validate and pass all quality gates.
+5. Q12 remains explicitly deferred by this spec revision; no hidden canary workflow is introduced indirectly.
 
 ---
 
@@ -337,20 +366,20 @@ Exit gate:
 - Risk: noisy knowledge retrieval overwhelms planner context.
   - Mitigation: strict limit/filter controls and deterministic top-K selection.
 
-- Risk: canary stage increases merge latency.
-  - Mitigation: lightweight canary profile and policy tuning.
+- Risk: existing auto-cleanup removes feature artifacts before Q10 can persist the data it needs.
+  - Mitigation: explicit merge-time summary persistence plus cleanup ordering rules that run only after graph artifacts are durable.
 
-- Risk: teams misinterpret blocked-after-canary as merge corruption.
-  - Mitigation: explicit status reason and rollback/fix-forward guidance artifact.
+- Risk: teams infer that Q12 still exists because of the legacy filename.
+  - Mitigation: this revision explicitly marks Q12 deferred and restricts all concrete implementation work in the document to Q10.
 
 ---
 
 ## 9. Definition of Done
 
-A completed KC milestone MUST include:
+A completed KG milestone MUST include:
 
 - changed files and rationale
 - policy/schema/tool catalog diffs
-- integration + parity tests
+- integration and parity tests
 - command verification summary
 - `spec-files/progress.md` update with residual tasks (if any)

package/spec-files/outstanding/agentic_orchestrator_observability_replay_spec.md

@@ -4,9 +4,9 @@
 > **Covers**: OID0 (full shared scaffolding) + OID1 (Q3 Replay Timeline) + Replay CLI surface
 > **Roadmap Mapping:** M33 / Q3
 
-**Version:** 1.0
-**Date:** 2026-03-05
-**Status:** Implementation-Ready
+**Version:** 1.1
+**Date:** 2026-03-15
+**Status:** Implemented in Repo (Retained Here Until Archived)
 
 ---