agentic-orchestrator 0.1.4 → 0.1.6

package/spec-files/outstanding/agentic_orchestrator_observability_integrity_diagnostics_spec.md

@@ -1,10 +1,10 @@
 # Feature Spec: Observability, Evidence Integrity, and Operational Diagnostics (AOP)
 
-> **Purpose of this document**: Define implementation-ready delivery for Q3, Q4, and Q11 from the Quality + Adoption execution roadmap: deterministic replay, tamper-evident evidence, and `aop doctor` readiness/drift diagnostics.
+> **Purpose of this document**: Provide implementation-ready delivery for Q3, Q4, and Q11 with an explicit architectural critique of the prior draft and a corrected execution plan aligned to the current control-plane/runtime architecture.
 
-**Version:** 1.0  
-**Date:** 2026-03-03  
-**Status:** Draft  
+**Version:** 2.0  
+**Date:** 2026-03-04  
+**Status:** Draft (Architecture-Corrected)  
 **Roadmap Mapping:** M33
 
 ---
@@ -14,80 +14,176 @@
 ### 0.1 Required Standards
 
 All implementation MUST preserve:
-- deterministic tool contracts and normalized error envelope
-- MCP/in-process parity for new tool paths
-- Nx + Vitest + architecture boundary validation
+- deterministic tool behavior and normalized error envelopes
+- MCP/in-process transport parity for all new tools
+- strict contract validation (`validate:mcp-contracts`) and architecture boundaries (`validate:architecture`)
 - runtime artifact ownership under `.aop/**`
+- existing idempotency semantics (`operation_id` replay/mismatch) for mutating tools
 
-### 0.2 Upstream Inputs
+### 0.2 Required Upstream Inputs
 
 Implementing agents MUST read:
-- `spec-files/agentic_orchestrator_spec.md`
-- `spec-files/agentic_orchestrator_mcp_formalization_spec.md`
-- `spec-files/progress.md`
+- `spec-files/outstanding/agentic_orchestrator_quality_adoption_execution_spec.md`
+- `apps/control-plane/src/mcp/tool-runtime.ts`
+- `apps/control-plane/src/mcp/operation-ledger.ts`
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/core/path-layout.ts`
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `apps/control-plane/src/core/gates.ts`
+- `apps/control-plane/src/application/services/reporting-service.ts`
+- `apps/control-plane/src/cli/types.ts`
+- `apps/control-plane/src/cli/cli-argument-parser.ts`
+- `apps/control-plane/src/cli/help-command-handler.ts`
+- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
+- `apps/control-plane/src/application/tools/tool-metadata.ts`
+- `agentic/orchestrator/defaults/policy.defaults.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/schemas/policy.user.schema.json`
+- `agentic/orchestrator/tools/catalog.json`
+- `apps/control-plane/scripts/validate-mcp-contracts.ts`
 
 ### 0.3 Feature Scope
 
 This spec implements:
-- **Q3** Deterministic Replay + Time-Travel Debugger
-- **Q4** Tamper-Evident Evidence Chain
-- **Q11** `aop doctor` Readiness + Drift Scanner
+- **Q3** Deterministic Replay + Time-Travel Diagnostics
+- **Q4** Tamper-Evident Evidence Integrity Chain
+- **Q11** `aop doctor` Readiness + Drift Diagnostics
+
+Out of scope:
+- autonomous RCA narrative generation
+- external KMS/HSM signing
+- remote telemetry backends
+- background daemonized doctor scans
 
 ---
 
-## 1. Objectives
+## 1. Architectural Critique of Prior Draft (v1.0)
 
-### 1.1 Must-Have Outcomes
+The prior draft was directionally correct but not implementation-safe against the current architecture.
 
-- Operators can reconstruct run behavior step-by-step with deterministic timeline artifacts.
-- Evidence artifacts become cryptographically tamper-evident.
-- Repos can be proactively checked for readiness and drift via `aop doctor`.
-- New capabilities are additive and feature-flagged where needed.
+1. **Replay design ignored existing idempotency ledger semantics**
+- Problem: timeline design treated every call as `ok/error` and did not model `operation_id` replay/mismatch outcomes.
+- Impact: operators cannot distinguish a real execution from a deterministic replay path.
+- Correction: timeline entries must include `idempotency_status` (`new|replay|mismatch|none`) and optionally `request_hash`.
 
-### 1.2 Non-Goals
+2. **Tool and CLI wiring was incomplete**
+- Problem: draft listed service files only, but current runtime requires full wiring through constants, kernel handlers, tool metadata, catalog, RBAC, docs generation, CLI enum/parser/help/bootstrap.
+- Impact: contract drift and startup/runtime failures.
+- Correction: include full tool/CLI lifecycle wiring as normative work.
 
-- No AI-generated RCA narratives in this phase (timeline + facts only).
-- No cryptographic key management/KMS integration in this phase (local signing primitives only).
-- No daemonized background doctor process; command is on-demand.
+3. **Evidence chain placement would break `evidence.latest`**
+- Problem: `GateService.evidenceLatest()` currently scans all `*.json` files by mtime in evidence dir.
+- Impact: adding `chain.json` or `manifest.json` can cause `evidence.latest` to return chain metadata instead of latest gate evidence.
+- Correction: constrain `evidenceLatest` to `gate-*.json` and `latest-*.json` only.
 
----
+4. **Evidence chain algorithm was insufficiently deterministic**
+- Problem: hash contract depended on ad hoc concatenation with timestamps and no algorithm versioning/canonical object format.
+- Impact: brittle verification and future incompatibility.
+- Correction: define canonical hash payload with explicit `algorithm_version` and stable field ordering.
 
-## 2. Architecture Decisions
+5. **Concurrency guarantees were underspecified**
+- Problem: replay sequence allocation and chain append were specified without lock strategy.
+- Impact: sequence collisions/interleaving corruption under concurrent tool calls and parallel gates.
+- Correction: require per-run replay lock and per-feature evidence-chain lock using existing file-lock primitives.
 
-### 2.1 Single Timeline Writer Contract
+6. **Doctor checks were too vague for deterministic implementation**
+- Problem: checks were listed by category but not by stable check IDs, severity rules, or deterministic ordering.
+- Impact: non-repeatable output and fragile tests.
+- Correction: define fixed check IDs, output ordering, and strict severity semantics.
 
-Timeline entries MUST be written by a dedicated service with append-only semantics and monotonic order. Call sites in tool runtime may emit events, but file persistence must go through one writer abstraction.
+7. **Feature gating model was missing from policy schema**
+- Problem: draft said “feature-flagged where needed” but provided no schema/default additions.
+- Impact: policy validation failure if teams add unmodeled fields.
+- Correction: add policy/default/schema contracts for observability and diagnostics toggles.
+
+8. **Rollout and retention interactions were not addressed**
+- Problem: no plan for replay file growth, no compatibility plan with evidence retention/pruning.
+- Impact: unbounded artifacts or false integrity failures.
+- Correction: add retention policy and explicit integrity-vs-pruning rules.
+
+---
 
-### 2.2 Evidence Chain Anchoring
+## 2. Revised Architecture Fit
 
-Every new gate/evidence artifact MUST be chained against the immediately prior chain hash for that feature to produce a deterministic tamper-evidence chain.
+### 2.1 Existing Runtime Touchpoints
 
-### 2.3 Doctor as Deterministic Scanner
+- **Tool execution boundary**: `ToolRuntime.callTool()` is the authoritative event point for replay timeline instrumentation (covers in-process and MCP equally).
+- **Idempotency source of truth**: `OperationLedger` already determines `new|replay|mismatch` for mutating operations.
+- **Evidence write path**: `GateService.gatesRun()` + `core/gates.ts` is the authoritative source for gate evidence artifacts.
+- **Read/report path**: `ReportingService` remains the read aggregation surface; integrity checks should not bypass service boundaries.
+- **CLI composition path**: command enum/parser/help/bootstrap require explicit updates for new command surfaces.
 
-`aop doctor` MUST run pure deterministic checks over local state/config/runtime assumptions and emit structured findings with severities and remediations.
+### 2.2 Boundary Rules (Normative)
+
+- `core/*` remains pure infrastructure/domain helpers; orchestration/policy decisions live in `application/services/*`.
+- replay timeline emission belongs in MCP runtime layer (`tool-runtime`) to guarantee transport parity.
+- evidence chain append/verify belongs in `application/services/*` and is invoked from `GateService` write path.
+- doctor scans are deterministic local checks only (file/config/runtime state), no network dependency.
+- all behavior toggles must be schema-backed and default-safe.
 
 ---
 
-## 3. Canonical Artifact Contracts
+## 3. Contracts and Schema Deltas
+
+## 3.1 Policy Extensions (Defaults + Schema + User Schema)
+
+Add policy sections:
+
+```yaml
+innovation:
+  replay_debugger: true
+  tamper_evident_evidence: false
+
+observability:
+  replay:
+    enabled: true
+    write_mode: best_effort     # best_effort | required
+    retention:
+      max_runs: 100
+      max_entries_per_run: 50000
+  evidence_integrity:
+    enabled: false
+    verify_on_read: false
+    algorithm_version: 1
+
+diagnostics:
+  doctor:
+    enabled: true
+    strict_fails_on: critical   # critical | warning
+```
+
+Normative behavior:
+- `innovation.replay_debugger` and `observability.replay.enabled` must both be true for timeline writes.
+- `innovation.tamper_evident_evidence` and `observability.evidence_integrity.enabled` must both be true for chain writes.
+- `write_mode=best_effort` logs failures but never fails tool execution; `required` fails the parent tool call.
+- doctor command remains callable; if `diagnostics.doctor.enabled=false`, it returns a report with a deterministic `doctor_disabled` info finding.
 
-### 3.1 Replay Timeline
+## 3.2 Replay Artifact Contracts
+
+Path:
 
 ```text
 .aop/runtime/replay/<run_id>.timeline.jsonl
+.aop/runtime/replay/<run_id>.meta.json
 ```
 
-Entry contract:
+Entry contract (`timeline.jsonl` line):
 
 ```json
 {
-  "ts": "2026-03-03T00:00:00Z",
+  "ts": "2026-03-04T00:00:00.000Z",
   "sequence": 42,
   "run_id": "run-123",
-  "feature_id": "my_feature",
-  "phase": "building",
-  "tool": "repo.apply_patch",
-  "operation_id": "op-abc",
-  "input_hash": "sha256:...",
+  "session_id": "session-abc",
+  "actor_type": "builder",
+  "feature_id": "feature_x",
+  "tool_name": "repo.apply_patch",
+  "mutating": true,
+  "operation_id": "repo_apply_patch__feature_x__uuid",
+  "idempotency_status": "new",
+  "request_hash": "sha256:...",
   "result": "ok",
   "error_code": null,
   "latency_ms": 317
@@ -95,280 +191,489 @@ Entry contract:
 ```
 
 Rules:
-- `sequence` MUST be strictly increasing within a run.
-- `input_hash` MUST be deterministic canonical JSON hash.
-- timeline file MUST remain append-only.
+- `sequence` strictly increases per `run_id`.
+- write path is append-only JSONL; no in-place mutation of prior lines.
+- `idempotency_status` must be one of `none|new|replay|mismatch`.
+- ordering for reads is by `sequence` ascending, never filesystem mtime.
+
+## 3.3 Evidence Integrity Contracts
 
-### 3.2 Evidence Chain
+Paths:
 
 ```text
-.aop/features/<feature_id>/evidence/chain.json
-.aop/features/<feature_id>/evidence/manifest.json
+.aop/features/<feature_id>/evidence/chain.jsonl
+.aop/features/<feature_id>/evidence/manifest.integrity.json
 ```
 
 Chain entry contract:
 
 ```json
 {
-  "artifact_path": "logs/gates/full/2026-03-03T10-15-00Z.log",
+  "algorithm_version": 1,
+  "sequence": 7,
+  "feature_id": "feature_x",
+  "artifact_rel_path": "evidence/gate-full-1741050000.json",
   "artifact_hash": "sha256:...",
   "prev_chain_hash": "sha256:...",
   "chain_hash": "sha256:...",
-  "created_at": "2026-03-03T10:15:01Z"
+  "created_at": "2026-03-04T10:15:01.000Z"
+}
+```
+
+Hash payload (canonical JSON, stable key order):
+
+```json
+{
+  "algorithm_version": 1,
+  "sequence": 7,
+  "feature_id": "feature_x",
+  "artifact_rel_path": "evidence/gate-full-1741050000.json",
+  "artifact_hash": "sha256:...",
+  "prev_chain_hash": "sha256:..."
 }
 ```
 
 Rules:
-- chain hash MUST include `artifact_hash + prev_chain_hash + artifact_path + created_at`.
-- verification MUST fail closed if artifact missing, hash mismatch, or chain discontinuity.
+- `chain_hash = sha256(canonical_json(hash_payload))`.
+- `manifest.integrity.json` stores `entry_count`, `head_hash`, `last_verified_at`, `last_verified_result`.
+- verification fails closed on missing artifact, hash mismatch, or chain discontinuity.
+- tracked artifacts in v1 scope: `gate-*.json`, `latest-*.json` only.
 
-### 3.3 Doctor Report
+## 3.4 Doctor Report Contract
+
+Path:
 
 ```text
 .aop/runtime/doctor/last_report.json
+.aop/runtime/doctor/history/<timestamp>.json
 ```
 
-Contract:
+Report contract:
 
 ```json
 {
   "ok": true,
-  "generated_at": "2026-03-03T10:30:00Z",
+  "generated_at": "2026-03-04T10:30:00.000Z",
   "summary": {
     "critical": 0,
-    "warning": 2,
-    "info": 5
+    "warning": 1,
+    "info": 3
   },
   "findings": [
     {
-      "id": "policy_gate_mismatch",
+      "id": "DRIFT_004_STALE_RUN_LEASE",
       "severity": "warning",
-      "message": "...",
-      "remediation": "...",
-      "refs": ["agentic/orchestrator/policy.yaml"]
+      "message": "Run lease appears stale and still populated",
+      "remediation": "Run `aop run --takeover-stale-run` or clear stale lease after verification",
+      "refs": [".aop/runtime/default/run-lease.json"]
     }
   ]
 }
 ```
 
----
+Rules:
+- findings sorted lexically by `id`.
+- summary counts derived strictly from findings array.
+- `ok = (critical===0)` in normal mode.
 
-## 4. Tool and CLI Surface
+## 3.5 Tool Surface Changes
 
-### 4.1 New MCP Tools
+### New Tools
 
-1. `replay.timeline_get`
-- Purpose: return timeline entries for a run with optional step/window filters.
-- Mutating: no.
+1. `replay.timeline_get` (read-only)
+- query timeline by `run_id` with optional `from_sequence`, `limit`, `feature_id`, `tool_name` filters.
 
-2. `evidence.verify_chain`
-- Purpose: verify evidence chain integrity for a feature.
-- Mutating: no.
+2. `evidence.verify_chain` (read-only)
+- verify integrity chain for a feature and return validity summary + first failure details.
 
-3. `doctor.run`
-- Purpose: execute all readiness/drift checks and persist report artifact.
-- Mutating: yes (writes report).
+3. `doctor.run` (mutating)
+- execute deterministic checks and persist doctor report artifacts.
 
-### 4.2 New CLI Commands
+### Required Wiring for Every New Tool
 
-- `aop replay --run-id <id> [--step N] [--window <count>]`
-- `aop evidence verify --feature-id <id>`
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/application/tools/tool-metadata.ts`
+- `agentic/orchestrator/tools/catalog.json`
+- `agentic/orchestrator/tools/schemas/input/*.json`
+- `agentic/orchestrator/tools/schemas/output/*.json`
+- `agentic/orchestrator/defaults/policy.defaults.yaml` RBAC lists
+- generated `agentic/orchestrator/tools.md`
+
+## 3.6 CLI Surface Changes
+
+Add commands:
+- `aop replay --run-id <id> [--from-sequence N] [--limit N] [--feature-id <id>] [--tool <name>] [--json]`
+- `aop evidence verify --feature-id <id> [--json]`
 - `aop doctor [--strict] [--json]`
 
-`--strict` contract:
-- exit non-zero if any `critical` finding exists.
+Required CLI wiring:
+- `apps/control-plane/src/cli/types.ts` (`CliCommand` additions + option fields)
+- `apps/control-plane/src/cli/cli-argument-parser.ts`
+- `apps/control-plane/src/cli/help-command-handler.ts`
+- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
+- new handlers under `apps/control-plane/src/cli/*-command-handler.ts`
+
+## 3.7 Error Code Additions
+
+Add to `apps/control-plane/src/core/error-codes.ts`:
+- `replay_run_not_found`
+- `replay_timeline_corrupt`
+- `replay_timeline_write_failed`
+- `evidence_chain_missing`
+- `evidence_chain_artifact_missing`
+- `evidence_chain_hash_mismatch`
+- `evidence_chain_discontinuity`
+- `evidence_integrity_write_failed`
+- `doctor_check_failed`
+- `doctor_report_write_failed`
 
 ---
 
-## 5. Detailed Feature Implementation
+## 4. Detailed Implementation Plan (Normative)
 
-## 5.1 Q3: Deterministic Replay + Time-Travel Debugger
+## OID0: Contract Scaffolding and Policy Wiring
 
-### 5.1.1 Implementation Targets
+### Goals
+- land schema/default/tool scaffolding first to prevent runtime/config drift
 
-- `apps/control-plane/src/application/services/replay-timeline-service.ts` (new)
-- `apps/control-plane/src/mcp/tool-runtime.ts` (timeline emit hooks)
-- `apps/control-plane/src/cli/replay-command-handler.ts` (new)
-- `apps/control-plane/src/cli/cli-argument-parser.ts`
+### File Targets
+- `agentic/orchestrator/defaults/policy.defaults.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/schemas/policy.user.schema.json`
+- `apps/control-plane/src/cli/init-command-handler.ts` (advanced policy template alignment)
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `agentic/orchestrator/tools/catalog.json`
 - `agentic/orchestrator/tools/schemas/input/replay.timeline_get.input.schema.json` (new)
 - `agentic/orchestrator/tools/schemas/output/replay.timeline_get.output.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/input/evidence.verify_chain.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/evidence.verify_chain.output.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/input/doctor.run.input.schema.json` (new)
+- `agentic/orchestrator/tools/schemas/output/doctor.run.output.schema.json` (new)
+- `agentic/orchestrator/tools.md`
+
+### Mandatory Checks
+- `npm run validate:mcp-contracts`
+- `npm run validate:architecture`
+
+Exit criteria:
+- policy composition validates with new fields
+- no contract validator drift
+
+---
+
+## OID1: Q3 Replay Timeline Foundation
+
+### Design
+
+Instrument `ToolRuntime` finalization path with deterministic timeline writes using a dedicated replay service.
+
+### Algorithm
+
+1. `ToolRuntime.callTool()` computes idempotency resolution.
+2. Construct replay event envelope after response is finalized (includes latency and result).
+3. `ReplayTimelineService.append(run_id, event)` acquires per-run lock.
+4. Service increments `sequence` from `<run_id>.meta.json`.
+5. Service appends JSONL line atomically and updates meta.
+6. `replay.timeline_get` reads JSONL, validates line shape, filters deterministically, sorts by sequence.
+
+### Implementation Targets
+
+- `apps/control-plane/src/application/services/replay-timeline-service.ts` (new)
+- `apps/control-plane/src/mcp/tool-runtime.ts`
+- `apps/control-plane/src/core/path-layout.ts` (new helpers for replay paths)
+- `apps/control-plane/src/core/kernel.ts` (tool handler registration and wrapper)
+
+### Behavioral Rules
 
-### 5.1.2 Algorithm
+- replay emission is best-effort by default; respects `observability.replay.write_mode`.
+- replay entries must be emitted for both success and failure responses.
+- `idempotency_status=replay` entries must still be written (diagnostic visibility).
 
-1. Tool runtime creates normalized call event envelope.
-2. Replay service computes canonical `input_hash`.
-3. Replay service obtains next `sequence` atomically.
-4. Entry appended to JSONL.
-5. `replay.timeline_get` reads, filters, and returns ordered entries.
+### Test Plan
 
-### 5.1.3 Error Codes
+- `apps/control-plane/test/replay-timeline-service.spec.ts` (new)
+- extend `apps/control-plane/test/tool-runtime.spec.ts` (idempotency status propagation)
+- extend `apps/control-plane/test/mcp.spec.ts` for `replay.timeline_get` parity
 
-- `run_not_found`
-- `replay_entry_invalid`
-- `timeline_corrupt`
+Exit criteria:
+- identical tool call sequence produces byte-identical timeline lines (except timestamps)
+- sequence strictly monotonic under concurrent writes
 
 ---
 
-## 5.2 Q4: Tamper-Evident Evidence Chain
+## OID2: Q4 Evidence Integrity Chain
 
-### 5.2.1 Implementation Targets
+### Design
 
-- `apps/control-plane/src/application/services/evidence-chain-service.ts` (new)
-- `apps/control-plane/src/application/services/gate-service.ts`
-- `apps/control-plane/src/cli/evidence-verify-command-handler.ts` (new)
-- `agentic/orchestrator/tools/schemas/input/evidence.verify_chain.input.schema.json` (new)
-- `agentic/orchestrator/tools/schemas/output/evidence.verify_chain.output.schema.json` (new)
+Add chain writer/validator service and integrate into gate evidence write path after final evidence JSON mutation.
 
-### 5.2.2 Write Path Integration
+### Critical Integration Correction
 
-On every evidence artifact write:
-1. calculate artifact hash
-2. load previous chain head
-3. compute new chain hash
-4. append chain entry
-5. update manifest metadata (entry count, head hash)
+Before chain integration, change `GateService.evidenceLatest()` selection logic:
+- include only files matching `gate-*.json` and `latest-*.json`
+- exclude `chain.jsonl`, `manifest.integrity.json`, and any non-gate evidence metadata
 
-### 5.2.3 Verify Path
+### Write Path Algorithm
 
-1. iterate chain entries in order
-2. recompute artifact hash and chain hash
-3. verify continuity
-4. emit result summary with first failing index if invalid
+1. Gate run writes evidence files (`core/gates.ts`).
+2. `GateService` applies incremental skipped-info rewrite if applicable.
+3. If evidence integrity enabled, `EvidenceIntegrityService.append(feature_id, artifact_path)` runs for each tracked artifact.
+4. Service computes artifact hash, reads head hash/sequence, computes next chain entry, appends to `chain.jsonl`, updates manifest.
 
-### 5.2.4 Error Codes
+### Verify Algorithm
 
-- `evidence_chain_missing`
-- `evidence_hash_mismatch`
-- `evidence_chain_discontinuity`
+1. Load chain entries in sequence order.
+2. For each entry, recompute artifact hash and chain hash from canonical payload.
+3. Check `prev_chain_hash` continuity.
+4. Return deterministic summary with first failing `sequence` and reason.
+
+### Implementation Targets
+
+- `apps/control-plane/src/application/services/evidence-integrity-service.ts` (new)
+- `apps/control-plane/src/application/services/gate-service.ts`
+- `apps/control-plane/src/core/path-layout.ts` (chain/manifest helpers)
+- `apps/control-plane/src/core/kernel.ts` (tool handler wiring)
+
+### Behavioral Rules
+
+- integrity chain is opt-in via policy toggles.
+- verification fails closed when artifacts are missing or corrupted.
+- chain verification is read-only and never mutates evidence artifacts.
+
+### Test Plan
+
+- `apps/control-plane/test/evidence-integrity-service.spec.ts` (new)
+- extend `apps/control-plane/test/gates.spec.ts`
+- add regression in `apps/control-plane/test/kernel.spec.ts` ensuring `evidence.latest` still returns gate evidence
+- extend `apps/control-plane/test/mcp.spec.ts` for `evidence.verify_chain` parity
+
+Exit criteria:
+- tampered artifact triggers deterministic failure reason
+- untampered chain verifies successfully
 
 ---
 
-## 5.3 Q11: `aop doctor` Readiness + Drift Scanner
+## OID3: Q11 Doctor Diagnostics
 
-### 5.3.1 Implementation Targets
+### Design
+
+Implement deterministic scanner with fixed check IDs and explicit severity mapping.
+
+### Required Checks (v1)
+
+- `DRIFT_001_POLICY_SCHEMA_INVALID` (critical)
+- `DRIFT_002_GATES_SCHEMA_INVALID` (critical)
+- `DRIFT_003_TOOL_CONTRACT_DRIFT` (warning)
+- `DRIFT_004_STALE_RUN_LEASE` (warning)
+- `DRIFT_005_RUNTIME_SESSION_ORPHANS` (warning)
+- `DRIFT_006_TIMELINE_CORRUPTION` (warning)
+- `DRIFT_007_EVIDENCE_CHAIN_INVALID` (warning when enabled)
+- `DRIFT_008_INDEX_STATE_INCONSISTENCY` (critical)
+
+### Deterministic Output Rules
+
+- check execution order fixed lexically by check ID
+- findings sorted by check ID
+- same repo state yields identical findings (except timestamp fields)
+
+### Implementation Targets
 
 - `apps/control-plane/src/application/services/doctor-service.ts` (new)
-- `apps/control-plane/src/cli/doctor-command-handler.ts` (new)
 - `agentic/orchestrator/schemas/doctor_report.schema.json` (new)
-- `agentic/orchestrator/tools/schemas/input/doctor.run.input.schema.json` (new)
-- `agentic/orchestrator/tools/schemas/output/doctor.run.output.schema.json` (new)
+- `apps/control-plane/src/core/path-layout.ts` (doctor report/history helpers)
+- `apps/control-plane/src/core/kernel.ts` (`doctor.run` handler)
 
-### 5.3.2 Scanner Categories
+### CLI Semantics
 
-1. **Environment checks**
-- node/npm/git availability
-- minimal tool versions
+- `aop doctor --strict`: non-zero exit when findings include severity at or above configured strict threshold.
+- `aop doctor --json`: emit raw report payload.
 
-2. **Config/schema checks**
-- policy/gates/agents schema-valid
-- MCP contract catalog consistency
+### Test Plan
 
-3. **Runtime health checks**
-- stale run lease anomalies
-- orphan runtime session mappings
-- corrupt/partial artifact detection
+- `apps/control-plane/test/doctor-service.spec.ts` (new)
+- extend `apps/control-plane/test/cli.unit.spec.ts` for parser/help/strict behavior
+- extend `apps/control-plane/test/mcp.spec.ts` for `doctor.run` parity
 
-4. **Drift checks**
-- policy-gates mismatch (required modes missing)
-- disabled mandatory invariants
+Exit criteria:
+- report artifact is schema-valid
+- strict mode exit behavior is deterministic and tested
 
-### 5.3.3 Severity Model
+---
 
-- `critical`: immediate run-blocking risk
-- `warning`: correctness/performance risk
-- `info`: advisory
+## OID4: Full Tool + CLI Surface Integration
 
-### 5.3.4 Error Codes
+### Goals
+- complete all runtime and CLI wiring for new commands/tools
 
-- `doctor_check_failed`
-- `doctor_report_invalid`
+### Tool Integration Targets
+
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/application/tools/tool-metadata.ts`
+- `agentic/orchestrator/tools/catalog.json`
+- `agentic/orchestrator/defaults/policy.defaults.yaml` RBAC sections
+- `agentic/orchestrator/tools.md`
+
+### CLI Integration Targets
+
+- `apps/control-plane/src/cli/types.ts`
+- `apps/control-plane/src/cli/cli-argument-parser.ts`
+- `apps/control-plane/src/cli/help-command-handler.ts`
+- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
+- `apps/control-plane/src/cli/replay-command-handler.ts` (new)
+- `apps/control-plane/src/cli/evidence-command-handler.ts` (new)
+- `apps/control-plane/src/cli/doctor-command-handler.ts` (new)
+
+### Mandatory Checks
+
+- `npm run validate:mcp-contracts`
+- `npm run typecheck`
+
+Exit criteria:
+- tools callable via both in-process and MCP clients
+- CLI commands documented and discoverable via `aop help`
 
 ---
 
-## 6. Milestones
+## OID5: Hardening, Performance, and Retention
+
+### Replay Retention
+
+Implement deterministic cleanup:
+- keep newest `max_runs` timeline files
+- cap each run by `max_entries_per_run` with rollover file strategy or truncation policy (must be deterministic and documented)
+
+### Evidence Integrity vs Retention Rule
+
+If a separate evidence pruning policy exists, integrity-tracked files must not be pruned silently.
+Required v1 behavior:
+- doctor emits `DRIFT_007_EVIDENCE_CHAIN_INVALID` if chain references missing files.
+- documentation explicitly states integrity verification requires preserving tracked artifacts.
 
-### OD1: Replay Foundation
+### Implementation Targets
 
-- timeline service + MCP tool + CLI command
-- end-to-end timeline generation for tool calls
+- `apps/control-plane/src/application/services/replay-timeline-service.ts`
+- `apps/control-plane/src/application/services/doctor-service.ts`
+- `README.md` (operational caveats)
 
-Exit gate:
-- deterministic ordering and stable hashes verified in tests
+### Test Plan
 
-### OD2: Evidence Integrity
+- retention behavior tests in replay service suite
+- doctor drift test for missing chained artifact
 
-- chain writer/validator integrated with gate artifacts
-- CLI + MCP verification paths added
+Exit criteria:
+- replay growth bounded by policy
+- integrity failures are diagnosable, not silent
 
-Exit gate:
-- tamper test fails correctly; untampered chain passes
+---
+
+## OID6: Rollout Strategy
+
+### Rollout Order
 
-### OD3: Doctor Diagnostics
+1. Ship OID0 scaffolding; all new toggles at safe defaults.
+2. Enable replay in best-effort mode in internal repos.
+3. Enable doctor reporting in warning mode.
+4. Enable evidence integrity for pilot repos with stable evidence retention.
+5. Optionally switch replay write mode to `required` for high-audit repos.
 
-- scanner categories implemented with schema-backed report
-- strict exit mode added
+### Rollback Rules
 
-Exit gate:
-- doctor report persisted and schema-valid across sample repos
+- Any critical runtime regression: disable feature toggles without schema rollback.
+- Keep artifacts backward-compatible; do not delete existing replay/chain files on disable.
 
 ---
 
-## 7. Test Strategy (Normative)
+## 5. Test Strategy (Normative)
 
-### 7.1 Unit Tests
+### 5.1 Unit Tests
 
-- replay sequence monotonicity
-- evidence chain hash correctness
-- doctor finding severity assignment
+- replay sequence allocation and line validation
+- idempotency status mapping (`new|replay|mismatch|none`)
+- chain hash recomputation and discontinuity detection
+- doctor check ID ordering and severity aggregation
 
-### 7.2 Integration Tests
+### 5.2 Integration Tests
 
-- real run produces replay entries for critical tool path
-- modified evidence file triggers chain verification failure
-- doctor surfaces intentionally broken policy/gate config
+- end-to-end run emits replay timeline entries for core tool path
+- evidence tamper triggers `evidence.verify_chain` failure
+- doctor identifies intentionally broken policy/schema/runtime artifacts
 
-### 7.3 Transport Parity
+### 5.3 Transport Parity
 
-- `replay.timeline_get` parity across in-process vs MCP
-- `evidence.verify_chain` parity across in-process vs MCP
-- `doctor.run` parity across in-process vs MCP
+- `replay.timeline_get` parity (in-process vs MCP)
+- `evidence.verify_chain` parity (in-process vs MCP)
+- `doctor.run` parity (in-process vs MCP)
 
-### 7.4 CLI Tests
+### 5.4 CLI Tests
 
-- parser + handler tests for `replay`, `evidence verify`, and `doctor`
-- strict mode exit-code behavior
+- parser/help/bootstrap coverage for `replay`, `evidence verify`, `doctor`
+- strict-mode exit behavior for doctor
+- JSON output mode correctness
+
+### 5.5 Regression Tests
+
+- `evidence.latest` unaffected by integrity metadata files
+- existing idempotency and gate behavior preserved
 
 ---
 
-## 8. Acceptance Criteria
+## 6. Acceptance Criteria
 
-1. Replay timeline exists and is queryable by run/step.
-2. Evidence chain verification reliably detects tampering.
-3. `aop doctor` returns deterministic structured findings and actionable remediations.
-4. New tools are fully contract-validated and RBAC-scoped.
-5. All quality gates pass (`lint`, `typecheck`, `test`, contract/architecture validation).
+1. Replay timeline is persisted under `.aop/runtime/replay` and queryable by `run_id` with deterministic ordering.
+2. Replay entries include idempotency status and distinguish replayed mutations from fresh execution.
+3. Evidence integrity chain detects tampering and discontinuities deterministically.
+4. `evidence.latest` remains stable and returns gate evidence (not integrity metadata).
+5. `aop doctor` produces deterministic, schema-valid findings with actionable remediations.
+6. New tools are fully wired (constants/kernel/metadata/catalog/schemas/RBAC/tools.md).
+7. CLI commands are fully wired (types/parser/help/bootstrap/handlers).
+8. Quality gates pass: `lint`, `typecheck`, `test`, `validate:mcp-contracts`, `validate:architecture`.
 
 ---
 
-## 9. Risks and Mitigations
+## 7. Core Tradeoffs
+
+1. **Replay coverage vs overhead**
+- More per-call fields increase diagnostic value but add write overhead.
+- Chosen tradeoff: lightweight JSONL entries with optional `request_hash`, default best-effort writes.
 
-- Risk: timeline file growth for long-running repos.
-  - Mitigation: deterministic retention and archive policy per run.
+2. **Integrity strictness vs operational flexibility**
+- Fail-closed verification improves audit trust but can break workflows under artifact pruning.
+- Chosen tradeoff: integrity is opt-in, with explicit doctor findings for retention drift.
 
-- Risk: false positives in doctor checks.
-  - Mitigation: finding IDs with explicit suppression config and rationale tracking.
+3. **Doctor breadth vs determinism**
+- Broad checks can produce noisy, environment-dependent output.
+- Chosen tradeoff: fixed local checks with stable IDs/severities, no external network probes.
 
-- Risk: performance overhead from hashing every artifact.
-  - Mitigation: stream hashing and incremental manifest update.
+4. **Command ergonomics vs parser simplicity**
+- Nested command (`aop evidence verify`) is user-friendly but requires explicit parser subcommand logic.
+- Chosen tradeoff: implement subcommand support explicitly rather than introducing ambiguous top-level aliases.
 
 ---
 
-## 10. Definition of Done
+## 8. Risks and Mitigations
+
+- Risk: timeline/chain file corruption due to concurrent writes.
+  - Mitigation: per-run and per-feature file-locking around append operations.
+
+- Risk: performance regression from per-call writes.
+  - Mitigation: append-only JSONL, bounded retention, best-effort mode by default.
+
+- Risk: false confidence from disabled/partial checks.
+  - Mitigation: doctor always reports explicit disabled-state finding IDs.
+
+- Risk: schema drift between defaults/user/schema/catalog.
+  - Mitigation: OID0-first delivery with mandatory `validate:mcp-contracts` and policy schema updates.
+
+---
 
-A session completing any OD milestone MUST provide:
-- changed files and rationale
-- schema/tool catalog deltas
-- test coverage additions
-- verification command results
-- `spec-files/progress.md` updates with next actionable tasks
+## 9. Definition of Done
 
+A completed implementation session MUST provide:
+- changed files and rationale by phase (OID0-OID6)
+- schema/catalog/RBAC deltas and regenerated `tools.md`
+- test additions and executed command results
+- rollback notes for toggles and operational caveats
+- `spec-files/progress.md` update entry with completed OID milestones and residual tasks