agentic-orchestrator 0.1.2 → 0.1.4

package/spec-files/outstanding/agentic_orchestrator_observability_integrity_diagnostics_spec.md

@@ -0,0 +1,374 @@
+# Feature Spec: Observability, Evidence Integrity, and Operational Diagnostics (AOP)
+
+> **Purpose of this document**: Define implementation-ready delivery for Q3, Q4, and Q11 from the Quality + Adoption execution roadmap: deterministic replay, tamper-evident evidence, and `aop doctor` readiness/drift diagnostics.
+
+**Version:** 1.0  
+**Date:** 2026-03-03  
+**Status:** Draft  
+**Roadmap Mapping:** M33
+
+---
+
+## 0. Standards and Dependencies
+
+### 0.1 Required Standards
+
+All implementation MUST preserve:
+- deterministic tool contracts and normalized error envelope
+- MCP/in-process parity for new tool paths
+- Nx + Vitest + architecture boundary validation
+- runtime artifact ownership under `.aop/**`
+
+### 0.2 Upstream Inputs
+
+Implementing agents MUST read:
+- `spec-files/agentic_orchestrator_spec.md`
+- `spec-files/agentic_orchestrator_mcp_formalization_spec.md`
+- `spec-files/progress.md`
+
+### 0.3 Feature Scope
+
+This spec implements:
+- **Q3** Deterministic Replay + Time-Travel Debugger
+- **Q4** Tamper-Evident Evidence Chain
+- **Q11** `aop doctor` Readiness + Drift Scanner
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- Operators can reconstruct run behavior step-by-step with deterministic timeline artifacts.
+- Evidence artifacts become cryptographically tamper-evident.
+- Repos can be proactively checked for readiness and drift via `aop doctor`.
+- New capabilities are additive and feature-flagged where needed.
+
+### 1.2 Non-Goals
+
+- No AI-generated RCA narratives in this phase (timeline + facts only).
+- No cryptographic key management/KMS integration in this phase (local signing primitives only).
+- No daemonized background doctor process; command is on-demand.
+
+---
+
+## 2. Architecture Decisions
+
+### 2.1 Single Timeline Writer Contract
+
+Timeline entries MUST be written by a dedicated service with append-only semantics and monotonic order. Call sites in tool runtime may emit events, but file persistence must go through one writer abstraction.
+
+### 2.2 Evidence Chain Anchoring
+
+Every new gate/evidence artifact MUST be chained against the immediately prior chain hash for that feature to produce a deterministic tamper-evidence chain.
+
+### 2.3 Doctor as Deterministic Scanner
+
+`aop doctor` MUST run pure deterministic checks over local state/config/runtime assumptions and emit structured findings with severities and remediations.
+
+---
+
+## 3. Canonical Artifact Contracts
+
+### 3.1 Replay Timeline
+
+```text
+.aop/runtime/replay/<run_id>.timeline.jsonl
+```
+
+Entry contract:
+
+```json
+{
+  "ts": "2026-03-03T00:00:00Z",
+  "sequence": 42,
+  "run_id": "run-123",
+  "feature_id": "my_feature",
+  "phase": "building",
+  "tool": "repo.apply_patch",
+  "operation_id": "op-abc",
+  "input_hash": "sha256:...",
+  "result": "ok",
+  "error_code": null,
+  "latency_ms": 317
+}
+```
+
+Rules:
+- `sequence` MUST be strictly increasing within a run.
+- `input_hash` MUST be deterministic canonical JSON hash.
+- timeline file MUST remain append-only.
+
+### 3.2 Evidence Chain
+
+```text
+.aop/features/<feature_id>/evidence/chain.json
+.aop/features/<feature_id>/evidence/manifest.json
+```
+
+Chain entry contract:
+
+```json
+{
+  "artifact_path": "logs/gates/full/2026-03-03T10-15-00Z.log",
+  "artifact_hash": "sha256:...",
+  "prev_chain_hash": "sha256:...",
+  "chain_hash": "sha256:...",
+  "created_at": "2026-03-03T10:15:01Z"
+}
+```
+
+Rules:
+- chain hash MUST include `artifact_hash + prev_chain_hash + artifact_path + created_at`.
+- verification MUST fail closed if artifact missing, hash mismatch, or chain discontinuity.
+
+### 3.3 Doctor Report
+
+```text
+.aop/runtime/doctor/last_report.json
+```
+
+Contract:
+
+```json
+{
+  "ok": true,
+  "generated_at": "2026-03-03T10:30:00Z",
+  "summary": {
+    "critical": 0,
+    "warning": 2,
+    "info": 5
+  },
+  "findings": [
+    {
+      "id": "policy_gate_mismatch",
+      "severity": "warning",
+      "message": "...",
+      "remediation": "...",
+      "refs": ["agentic/orchestrator/policy.yaml"]
+    }
+  ]
+}
+```
+
+---
+
+## 4. Tool and CLI Surface
+
+### 4.1 New MCP Tools
+
+1. `replay.timeline_get`
+- Purpose: return timeline entries for a run with optional step/window filters.
+- Mutating: no.
+
+2. `evidence.verify_chain`
+- Purpose: verify evidence chain integrity for a feature.
+- Mutating: no.
+
+3. `doctor.run`
+- Purpose: execute all readiness/drift checks and persist report artifact.
+- Mutating: yes (writes report).
+
+### 4.2 New CLI Commands
+
+- `aop replay --run-id <id> [--step N] [--window <count>]`
+- `aop evidence verify --feature-id <id>`
+- `aop doctor [--strict] [--json]`
+
+`--strict` contract:
+- exit non-zero if any `critical` finding exists.
+
+---
+
+## 5. Detailed Feature Implementation
+
+## 5.1 Q3: Deterministic Replay + Time-Travel Debugger
+
+### 5.1.1 Implementation Targets
+
+- `apps/control-plane/src/application/services/replay-timeline-service.ts` (new)
+- `apps/control-plane/src/mcp/tool-runtime.ts` (timeline emit hooks)
+- `apps/control-plane/src/cli/replay-command-handler.ts` (new)
+- `apps/control-plane/src/cli/cli-argument-parser.ts`
+- `agentic/orchestrator/tools/schemas/input/replay.timeline_get.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/replay.timeline_get.output.schema.json` (new)
+
+### 5.1.2 Algorithm
+
+1. Tool runtime creates normalized call event envelope.
+2. Replay service computes canonical `input_hash`.
+3. Replay service obtains next `sequence` atomically.
+4. Entry appended to JSONL.
+5. `replay.timeline_get` reads, filters, and returns ordered entries.
+
+### 5.1.3 Error Codes
+
+- `run_not_found`
+- `replay_entry_invalid`
+- `timeline_corrupt`
+
+---
+
+## 5.2 Q4: Tamper-Evident Evidence Chain
+
+### 5.2.1 Implementation Targets
+
+- `apps/control-plane/src/application/services/evidence-chain-service.ts` (new)
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `apps/control-plane/src/cli/evidence-verify-command-handler.ts` (new)
+- `agentic/orchestrator/tools/schemas/input/evidence.verify_chain.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/evidence.verify_chain.output.schema.json` (new)
+
+### 5.2.2 Write Path Integration
+
+On every evidence artifact write:
+1. calculate artifact hash
+2. load previous chain head
+3. compute new chain hash
+4. append chain entry
+5. update manifest metadata (entry count, head hash)
+
+### 5.2.3 Verify Path
+
+1. iterate chain entries in order
+2. recompute artifact hash and chain hash
+3. verify continuity
+4. emit result summary with first failing index if invalid
+
+### 5.2.4 Error Codes
+
+- `evidence_chain_missing`
+- `evidence_hash_mismatch`
+- `evidence_chain_discontinuity`
+
+---
+
+## 5.3 Q11: `aop doctor` Readiness + Drift Scanner
+
+### 5.3.1 Implementation Targets
+
+- `apps/control-plane/src/application/services/doctor-service.ts` (new)
+- `apps/control-plane/src/cli/doctor-command-handler.ts` (new)
+- `agentic/orchestrator/schemas/doctor_report.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/input/doctor.run.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/doctor.run.output.schema.json` (new)
+
+### 5.3.2 Scanner Categories
+
+1. **Environment checks**
+- node/npm/git availability
+- minimal tool versions
+
+2. **Config/schema checks**
+- policy/gates/agents schema-valid
+- MCP contract catalog consistency
+
+3. **Runtime health checks**
+- stale run lease anomalies
+- orphan runtime session mappings
+- corrupt/partial artifact detection
+
+4. **Drift checks**
+- policy-gates mismatch (required modes missing)
+- disabled mandatory invariants
+
+### 5.3.3 Severity Model
+
+- `critical`: immediate run-blocking risk
+- `warning`: correctness/performance risk
+- `info`: advisory
+
+### 5.3.4 Error Codes
+
+- `doctor_check_failed`
+- `doctor_report_invalid`
+
+---
+
+## 6. Milestones
+
+### OD1: Replay Foundation
+
+- timeline service + MCP tool + CLI command
+- end-to-end timeline generation for tool calls
+
+Exit gate:
+- deterministic ordering and stable hashes verified in tests
+
+### OD2: Evidence Integrity
+
+- chain writer/validator integrated with gate artifacts
+- CLI + MCP verification paths added
+
+Exit gate:
+- tamper test fails correctly; untampered chain passes
+
+### OD3: Doctor Diagnostics
+
+- scanner categories implemented with schema-backed report
+- strict exit mode added
+
+Exit gate:
+- doctor report persisted and schema-valid across sample repos
+
+---
+
+## 7. Test Strategy (Normative)
+
+### 7.1 Unit Tests
+
+- replay sequence monotonicity
+- evidence chain hash correctness
+- doctor finding severity assignment
+
+### 7.2 Integration Tests
+
+- real run produces replay entries for critical tool path
+- modified evidence file triggers chain verification failure
+- doctor surfaces intentionally broken policy/gate config
+
+### 7.3 Transport Parity
+
+- `replay.timeline_get` parity across in-process vs MCP
+- `evidence.verify_chain` parity across in-process vs MCP
+- `doctor.run` parity across in-process vs MCP
+
+### 7.4 CLI Tests
+
+- parser + handler tests for `replay`, `evidence verify`, and `doctor`
+- strict mode exit-code behavior
+
+---
+
+## 8. Acceptance Criteria
+
+1. Replay timeline exists and is queryable by run/step.
+2. Evidence chain verification reliably detects tampering.
+3. `aop doctor` returns deterministic structured findings and actionable remediations.
+4. New tools are fully contract-validated and RBAC-scoped.
+5. All quality gates pass (`lint`, `typecheck`, `test`, contract/architecture validation).
+
+---
+
+## 9. Risks and Mitigations
+
+- Risk: timeline file growth for long-running repos.
+  - Mitigation: deterministic retention and archive policy per run.
+
+- Risk: false positives in doctor checks.
+  - Mitigation: finding IDs with explicit suppression config and rationale tracking.
+
+- Risk: performance overhead from hashing every artifact.
+  - Mitigation: stream hashing and incremental manifest update.
+
+---
+
+## 10. Definition of Done
+
+A session completing any OD milestone MUST provide:
+- changed files and rationale
+- schema/tool catalog deltas
+- test coverage additions
+- verification command results
+- `spec-files/progress.md` updates with next actionable tasks
+