agentic-orchestrator 0.1.2 → 0.1.4

package/spec-files/outstanding/agentic_orchestrator_planning_review_quality_spec.md

@@ -0,0 +1,466 @@
+# Feature Spec: Planning and Review Quality Hardening (AOP)
+
+> **Purpose of this document**: Define implementation-ready delivery for Q1, Q2, Q7, Q8, and Q9: semantic collision detection, feasibility scoring, prompt contract tests, flaky gate intelligence, and generated review briefs.
+
+**Version:** 1.0  
+**Date:** 2026-03-03  
+**Status:** Draft  
+**Roadmap Mapping:** M34-M36
+
+---
+
+## 0. Standards and Scope
+
+### 0.1 Standards
+
+All implementation MUST preserve:
+- deterministic plan admission behavior
+- policy/schema governance
+- transport parity for new tools
+- existing collision + lock safety guarantees
+
+### 0.2 Feature Scope
+
+This spec implements:
+- **Q1** Semantic Collision Engine
+- **Q2** Plan Feasibility Scoring
+- **Q7** Prompt Contract Test Harness
+- **Q8** Flaky Gate Intelligence + Quarantine Workflow
+- **Q9** Auto-Generated Review Briefs
+
+### 0.3 Out of Scope
+
+- ML-based probabilistic planning heuristics
+- autonomous prompt rewriting
+- automatic flaky-test suppression without accountability metadata
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- detect semantic collisions before expensive build/QA loops
+- prevent low-quality plans from advancing when policy requires score threshold
+- protect planner/builder/qa behavior from prompt regressions
+- reduce gate noise caused by flaky tests
+- provide deterministic, high-signal review briefs for human approval
+
+### 1.2 Non-Goals
+
+- no change to explicit user merge control semantics
+- no replacement of current gate profiles
+- no dynamic prompt evaluation service
+
+---
+
+## 2. Architecture Decisions
+
+### 2.1 Collision Checks as Layered Validation
+
+Plan admission remains layered:
+1. schema validation
+2. policy validation
+3. path/contract collision checks
+4. **semantic collision checks (new)**
+5. feasibility scoring threshold check (optional)
+
+### 2.2 Prompt Contracts as CI Artifact
+
+Prompt contracts are treated as repository source contracts with test fixtures and invariants. They are verified by deterministic scripts in CI, not by runtime agent heuristics.
+
+### 2.3 Flaky Intelligence is Advisory + Governed
+
+Flaky classification influences reporting and quarantine workflow but MUST NOT silently bypass required merge gates without explicit policy.
+
+### 2.4 Review Briefs are Deterministic Summaries
+
+Review briefs MUST be generated from canonical artifacts only (plan, state, diff, gate evidence, collisions, locks), not free-form model synthesis.
+
+---
+
+## 3. Contracts and Schema Deltas
+
+### 3.1 Policy Extensions
+
+```yaml
+innovation:
+  semantic_collisions: false
+  plan_feasibility_scoring: false
+  prompt_contract_tests: false
+  flaky_gate_intelligence: false
+  review_briefs: false
+
+planning:
+  feasibility:
+    enabled: false
+    minimum_score: 70
+    weights:
+      scope_realism: 0.25
+      test_sufficiency: 0.25
+      dependency_completeness: 0.2
+      contract_risk: 0.15
+      lock_contention_risk: 0.15
+
+testing:
+  flaky:
+    enabled: false
+    quarantine:
+      max_days: 14
+      require_owner: true
+      require_reason: true
+```
+
+### 3.2 New/Extended Artifacts
+
+```text
+.aop/
+  analytics/flaky-tests.json
+  features/<feature_id>/review_brief.json
+
+agentic/orchestrator/
+  prompt-contract/
+    fixtures/
+    invariants/
+  schemas/review_brief.schema.json
+```
+
+### 3.3 Tool Surface Additions
+
+- `plan.score` (read-only)
+- `gate.flaky_report_get` (read-only)
+- `report.feature_review_brief` (read-only)
+
+`collisions.scan` output schema must be extended for `semantic_conflicts[]`.
+
+---
+
+## 4. Detailed Feature Specs
+
+## 4.1 Q1: Semantic Collision Engine
+
+### 4.1.1 Problem
+Two features can avoid path collisions yet conflict semantically (same API operation evolution, incompatible event payloads, conflicting migration intent).
+
+### 4.1.2 Design
+Add semantic analyzers executed during `plan.submit` and `collisions.scan` when enabled.
+
+### 4.1.3 Analyzer Modules
+
+- `OpenApiSemanticAnalyzer`
+- `EventSchemaSemanticAnalyzer`
+- `DbIntentSemanticAnalyzer`
+
+### 4.1.4 Output Contract
+
+```json
+{
+  "semantic_conflicts": [
+    {
+      "type": "openapi_operation_conflict",
+      "feature_a": "feature_1",
+      "feature_b": "feature_2",
+      "fingerprint": "sha256:...",
+      "severity": "high",
+      "details": {
+        "path": "/users/{id}",
+        "method": "PATCH"
+      }
+    }
+  ]
+}
+```
+
+### 4.1.5 File Targets
+
+- `apps/control-plane/src/application/services/semantic-collision-service.ts` (new)
+- `apps/control-plane/src/application/services/plan-service.ts`
+- `apps/control-plane/src/application/services/reporting-service.ts`
+- `agentic/orchestrator/tools/schemas/output/collisions.scan.output.schema.json`
+
+### 4.1.6 Acceptance Criteria
+
+- semantic conflicts produce stable fingerprints and deterministic ordering
+- policy `collision_policy` handling applies to semantic conflicts identically to existing collisions
+
+---
+
+## 4.2 Q2: Plan Feasibility Scoring
+
+### 4.2.1 Problem
+Schema-valid plans can still be operationally poor and drive retries.
+
+### 4.2.2 Design
+Add deterministic scoring service and optional threshold gate in plan admission.
+
+### 4.2.3 Scoring Components
+
+- scope realism: file breadth vs stated summary/criteria
+- test sufficiency: acceptance criteria mapped to gate/test obligations
+- dependency completeness: missing/unsatisfied dependencies
+- contract risk: openapi/events/db impact complexity
+- lock contention risk: likelihood of blocking based on held/pending locks
+
+### 4.2.4 Tool Contract (`plan.score`)
+
+```json
+{
+  "feature_id": "my_feature",
+  "score": 78,
+  "breakdown": {
+    "scope_realism": 0.8,
+    "test_sufficiency": 0.7,
+    "dependency_completeness": 0.9,
+    "contract_risk": 0.6,
+    "lock_contention_risk": 0.7
+  },
+  "reasons": [
+    "Acceptance criteria #3 has no mapped verification target"
+  ]
+}
+```
+
+### 4.2.5 File Targets
+
+- `apps/control-plane/src/application/services/plan-feasibility-service.ts` (new)
+- `apps/control-plane/src/application/services/plan-service.ts`
+- `agentic/orchestrator/tools/schemas/output/plan.score.output.schema.json` (new)
+
+### 4.2.6 Acceptance Criteria
+
+- score for same inputs is stable across repeated calls
+- below-threshold plans blocked only when feasibility gate is enabled
+
+---
+
+## 4.3 Q7: Prompt Contract Test Harness
+
+### 4.3.1 Problem
+Prompt edits currently risk behavior drift without dedicated regression coverage.
+
+### 4.3.2 Design
+Fixture + invariant harness validates role outputs against deterministic expectations.
+
+### 4.3.3 Invariant Set (Initial)
+
+- planner output must include actionable plan artifacts
+- builder output must not propose forbidden tool usage
+- QA output must contain explicit evidence/test update intent
+
+### 4.3.4 Tooling
+
+- script: `apps/control-plane/scripts/test-prompt-contracts.mjs`
+- repo assets:
+  - `agentic/orchestrator/prompt-contract/fixtures/*`
+  - `agentic/orchestrator/prompt-contract/invariants/*`
+- package script:
+  - `npm run test:prompt-contracts`
+
+### 4.3.5 CI Contract
+
+If prompt files under `agentic/orchestrator/prompts/` change, prompt-contract suite MUST run and block merge on failure.
+
+### 4.3.6 Acceptance Criteria
+
+- invariant failures produce explicit diff-like diagnostics
+- prompt harness artifacts persist under `.aop/runtime/prompt-contract/`
+
+---
+
+## 4.4 Q8: Flaky Gate Intelligence + Quarantine Workflow
+
+### 4.4.1 Problem
+Flaky failures create noisy retries and reduce confidence in gate results.
+
+### 4.4.2 Design
+Track per-test stability history and expose deterministic suspect scoring and quarantine governance.
+
+### 4.4.3 Flake Model (Deterministic)
+
+Per test key `<framework>:<suite>:<test_name>`:
+- rolling window results
+- failure-rate and alternation index
+- quarantine metadata (`owner`, `reason`, `expires_at`)
+
+### 4.4.4 Quarantine Rules
+
+- quarantine entry requires owner + reason
+- quarantine expires automatically
+- expired quarantine generates reminder finding/event
+- quarantine cannot silently satisfy required merge gates unless policy explicitly permits
+
+### 4.4.5 File Targets
+
+- `apps/control-plane/src/application/services/flaky-gate-service.ts` (new)
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `agentic/orchestrator/tools/schemas/output/gate.flaky_report_get.output.schema.json` (new)
+
+### 4.4.6 Acceptance Criteria
+
+- flaky suspect list deterministic for same history input
+- quarantine lifecycle enforced by policy constraints
+
+---
+
+## 4.5 Q9: Auto-Generated Review Briefs
+
+### 4.5.1 Problem
+Human reviewers must manually synthesize plan/diff/evidence context.
+
+### 4.5.2 Design
+Generate concise structured review brief artifacts for each reviewable feature.
+
+### 4.5.3 Brief Sections
+
+- intent and scope summary
+- high-risk paths/contracts
+- gate outcome matrix
+- unresolved risks/questions
+- evidence references
+
+### 4.5.4 Artifact Contract
+
+```json
+{
+  "feature_id": "my_feature",
+  "intent_summary": "Add idempotent billing retry guard",
+  "high_risk_paths": ["libs/billing/retry.ts"],
+  "contract_changes": {
+    "openapi": false,
+    "events": true,
+    "db": false
+  },
+  "gate_summary": {
+    "fast": "pass",
+    "full": "pass",
+    "merge": "na"
+  },
+  "open_questions": [
+    "Should retry cap be environment-specific?"
+  ],
+  "evidence_refs": [
+    ".aop/features/my_feature/evidence/gates/full-2026-03-03.json"
+  ]
+}
+```
+
+### 4.5.5 File Targets
+
+- `apps/control-plane/src/application/services/review-brief-service.ts` (new)
+- `apps/control-plane/src/application/services/reporting-service.ts`
+- `packages/web-dashboard/src/components/ReviewPanel.tsx`
+- `agentic/orchestrator/schemas/review_brief.schema.json` (new)
+
+### 4.5.6 Acceptance Criteria
+
+- review brief generated for `ready_to_merge` features
+- brief content deterministic and sorted consistently
+
+---
+
+## 5. Milestones
+
+### PRQ1: Semantic + Feasibility Admission
+
+Tasks:
+1. Implement semantic analyzers + schema updates.
+2. Implement feasibility scoring + threshold enforcement.
+
+Exit gate:
+- conflicting and low-feasibility plans blocked deterministically when enabled.
+
+### PRQ2: Prompt Contract Reliability
+
+Tasks:
+1. Add fixtures/invariants and harness script.
+2. Wire CI enforcement for prompt changes.
+
+Exit gate:
+- prompt regressions are caught in CI with clear diagnostics.
+
+### PRQ3: Flaky Gate Intelligence
+
+Tasks:
+1. Add flake analytics persistence and report tool.
+2. Add quarantine governance and expiry checks.
+
+Exit gate:
+- flaky signal exposed without weakening mandatory gates by default.
+
+### PRQ4: Review Brief Automation
+
+Tasks:
+1. Implement review-brief service and artifact generation.
+2. Add MCP report tool and dashboard integration.
+
+Exit gate:
+- reviewers receive deterministic brief before merge approval.
+
+---
+
+## 6. Test Strategy (Normative)
+
+### 6.1 Unit Tests
+
+- each semantic analyzer rule
+- score determinism and weight normalization
+- prompt invariant validators
+- flaky score and quarantine expiry logic
+- brief assembly ordering and content extraction
+
+### 6.2 Integration Tests
+
+- `plan.submit` blocked by semantic conflict
+- `plan.submit` blocked by low feasibility when enabled
+- prompt-contract harness pass/fail suites
+- flaky report generation from realistic gate histories
+- review brief generation from live feature artifacts
+
+### 6.3 Transport Parity Tests
+
+- `plan.score`
+- `gate.flaky_report_get`
+- `report.feature_review_brief`
+
+### 6.4 UI Tests
+
+- review panel displays brief sections and evidence links
+- no brief available state handled clearly
+
+---
+
+## 7. Acceptance Criteria
+
+1. Q1/Q2 quality gates work deterministically and are policy-controlled.
+2. Prompt contract harness is operational and CI-gated.
+3. Flaky intelligence improves signal while preserving gate integrity.
+4. Review briefs are generated and consumable in CLI/dashboard flows.
+5. All validation gates pass (`lint`, `typecheck`, `test`, contract/architecture validators).
+
+---
+
+## 8. Risks and Mitigations
+
+- Risk: semantic analyzers produce false positives.
+  - Mitigation: confidence/severity labels + precise conflict details and opt-in rollout.
+
+- Risk: scoring model over-rejects valid plans.
+  - Mitigation: threshold configurable and disabled by default.
+
+- Risk: prompt fixtures become stale with runtime evolution.
+  - Mitigation: fixture versioning and required update notes on role contract changes.
+
+- Risk: quarantine becomes an abuse path.
+  - Mitigation: owner/reason/expiry mandatory and surfaced in status/reporting.
+
+---
+
+## 9. Definition of Done
+
+A completed PRQ milestone MUST include:
+- changed file inventory
+- schema/catalog updates
+- new tests and coverage notes
+- verification command summaries
+- `spec-files/progress.md` task closure and next steps
+

package/spec-files/outstanding/agentic_orchestrator_quality_adoption_execution_spec.md

@@ -0,0 +1,198 @@
+# Feature Spec: Quality + Adoption Execution Master Index (AOP)
+
+> **Purpose of this document**: Serve as the orchestration index for split implementation specs covering Q1-Q12. Detailed implementation requirements live in the feature-set specs listed below.
+
+**Version:** 1.1  
+**Date:** 2026-03-03  
+**Status:** Draft (Split-Spec Edition)  
+**Roadmap Window:** M33-M38
+
+---
+
+## 1. Why This Split Exists
+
+The original monolithic execution document covered too many orthogonal concerns in one spec. It is now split into focused specs by feature set so future agents can execute with clearer boundaries, lower context cost, and cleaner milestone ownership.
+
+---
+
+## 2. Canonical Split Specs
+
+### 2.1 Observability/Integrity/Diagnostics (M33)
+
+- `spec-files/agentic_orchestrator_observability_integrity_diagnostics_spec.md`
+
+Implements:
+- Q3 Deterministic Replay + Time-Travel Debugger
+- Q4 Tamper-Evident Evidence Chain
+- Q11 `aop doctor` Readiness + Drift Scanner
+
+### 2.2 Planning/Review Quality Hardening (M34-M36)
+
+- `spec-files/agentic_orchestrator_planning_review_quality_spec.md`
+
+Implements:
+- Q1 Semantic Collision Engine
+- Q2 Plan Feasibility Scoring
+- Q7 Prompt Contract Test Harness
+- Q8 Flaky Gate Intelligence + Quarantine Workflow
+- Q9 Auto-Generated Review Briefs
+
+### 2.3 Enterprise Governance + Secure Dashboard (M37)
+
+- `spec-files/agentic_orchestrator_enterprise_governance_dashboard_spec.md`
+
+Implements:
+- Q5 Compliance Policy Packs + Control Export
+- Q6 Dashboard AuthN/AuthZ + SSO + Audit Federation
+
+### 2.4 Knowledge + Progressive Merge Safety (M38)
+
+- `spec-files/agentic_orchestrator_knowledge_canary_spec.md`
+
+Implements:
+- Q10 Cross-Feature Knowledge Graph
+- Q12 Progressive Merge Guardrails (Canary Verification)
+
+---
+
+## 3. Feature Outcome Matrix
+
+| ID | Feature | Primary Outcome | Detailed Spec |
+|---|---|---|---|
+| Q1 | Semantic Collision Engine | Fewer hidden integration conflicts before build/QA | `agentic_orchestrator_planning_review_quality_spec.md` |
+| Q2 | Plan Feasibility Scoring | Early detection of weak plans | `agentic_orchestrator_planning_review_quality_spec.md` |
+| Q3 | Replay + Time-Travel | Faster incident RCA | `agentic_orchestrator_observability_integrity_diagnostics_spec.md` |
+| Q4 | Tamper-Evident Evidence | Stronger audit trust | `agentic_orchestrator_observability_integrity_diagnostics_spec.md` |
+| Q5 | Compliance Packs + Export | Faster enterprise onboarding | `agentic_orchestrator_enterprise_governance_dashboard_spec.md` |
+| Q6 | Dashboard Auth + SSO + Audit | Production-safe dashboard operation | `agentic_orchestrator_enterprise_governance_dashboard_spec.md` |
+| Q7 | Prompt Contract Tests | Prompt regression prevention | `agentic_orchestrator_planning_review_quality_spec.md` |
+| Q8 | Flaky Intelligence | Higher gate signal quality | `agentic_orchestrator_planning_review_quality_spec.md` |
+| Q9 | Review Briefs | Faster/high-quality human review | `agentic_orchestrator_planning_review_quality_spec.md` |
+| Q10 | Knowledge Graph | Better cross-feature reuse | `agentic_orchestrator_knowledge_canary_spec.md` |
+| Q11 | `aop doctor` | Faster readiness/drift triage | `agentic_orchestrator_observability_integrity_diagnostics_spec.md` |
+| Q12 | Canary Merge Verification | Safer merge promotion | `agentic_orchestrator_knowledge_canary_spec.md` |
+
+---
+
+## 4. Cross-Spec Coordination Rules
+
+### 4.1 Shared Constraints (Normative)
+
+All split specs MUST preserve:
+- deterministic MCP contract behavior
+- explicit user approval for merge promotion
+- policy/schema-driven validation
+- `.aop/**` runtime artifact ownership
+- architecture boundary validation and transport parity tests
+
+### 4.2 Shared Policy Namespace
+
+All new toggles MUST be grouped under:
+
+```yaml
+innovation:
+  semantic_collisions: false
+  plan_feasibility_scoring: false
+  replay_debugger: true
+  tamper_evident_evidence: false
+  compliance_packs: false
+  dashboard_auth: false
+  prompt_contract_tests: false
+  flaky_gate_intelligence: false
+  review_briefs: false
+  knowledge_graph: false
+  canary_merge_verification: false
+```
+
+### 4.3 Tool Contract Integration
+
+Any new tool introduced in a child spec MUST include:
+- catalog entry
+- input/output schemas
+- RBAC entry
+- tool docs regeneration
+- parity tests (`inprocess` vs `mcp`)
+
+---
+
+## 5. Milestone Ordering and Dependencies
+
+### M33 (must complete first)
+
+From `observability_integrity_diagnostics_spec`:
+- replay timeline
+- evidence chain verification
+- doctor scanner
+
+Dependency rationale:
+- provides baseline observability needed to debug and harden later milestones
+
+### M34-M36
+
+From `planning_review_quality_spec`:
+- semantic collisions + feasibility scoring
+- prompt contracts
+- flaky intelligence
+- review briefs
+
+Dependency rationale:
+- quality-admission and review optimizations depend on M33 observability artifacts
+
+### M37
+
+From `enterprise_governance_dashboard_spec`:
+- compliance export
+- dashboard auth/authz/audit federation
+
+Dependency rationale:
+- enterprise controls consume stable evidence/reporting from prior milestones
+
+### M38
+
+From `knowledge_canary_spec`:
+- knowledge graph retrieval
+- canary merge verification
+
+Dependency rationale:
+- canary and graph quality improve with historical artifacts generated by M33-M37
+
+---
+
+## 6. Execution Workflow for Future Agents
+
+For each milestone:
+
+1. Read this master index and only the child spec for the milestone in scope.
+2. Implement file-level targets from that child spec.
+3. Add/adjust contract artifacts (catalog/schemas/policy/docs).
+4. Add tests per child-spec test plan.
+5. Run required gates:
+   - `npm run lint`
+   - `npm run typecheck`
+   - `npm test`
+   - `npm run validate:mcp-contracts`
+   - `npm run validate:architecture`
+6. Update `spec-files/progress.md` with:
+   - completed tasks
+   - residual next tasks
+   - blockers/risk notes
+
+---
+
+## 7. Success Metrics (Program-Level)
+
+- 30% reduction in failed merges due to late conflict discovery.
+- 40% reduction in incident triage time.
+- 50% reduction in flaky failure noise within 60 days.
+- enterprise security/review onboarding reduced to <2 weeks for pilot rollout.
+- 20% faster reviewer approval turnaround.
+
+---
+
+## 8. Definition of Done (Master)
+
+The split initiative is complete only when:
+1. Q1-Q12 are delivered via their child specs.
+2. all feature flags and schema/tool contracts are integrated and validated.
+3. no regressions occur in existing deterministic behavior.
+4. `spec-files/progress.md` marks all child-spec milestones complete.