agentic-orchestrator 0.1.6 → 0.1.8

package/spec-files/outstanding/agentic_orchestrator_observability_integrity_diagnostics_spec.md

@@ -14,6 +14,7 @@
 ### 0.1 Required Standards
 
 All implementation MUST preserve:
+
 - deterministic tool behavior and normalized error envelopes
 - MCP/in-process transport parity for all new tools
 - strict contract validation (`validate:mcp-contracts`) and architecture boundaries (`validate:architecture`)
@@ -23,6 +24,7 @@ All implementation MUST preserve:
 ### 0.2 Required Upstream Inputs
 
 Implementing agents MUST read:
+
 - `spec-files/outstanding/agentic_orchestrator_quality_adoption_execution_spec.md`
 - `apps/control-plane/src/mcp/tool-runtime.ts`
 - `apps/control-plane/src/mcp/operation-ledger.ts`
@@ -47,11 +49,13 @@ Implementing agents MUST read:
 ### 0.3 Feature Scope
 
 This spec implements:
+
 - **Q3** Deterministic Replay + Time-Travel Diagnostics
 - **Q4** Tamper-Evident Evidence Integrity Chain
 - **Q11** `aop doctor` Readiness + Drift Diagnostics
 
 Out of scope:
+
 - autonomous RCA narrative generation
 - external KMS/HSM signing
 - remote telemetry backends
@@ -64,41 +68,49 @@ Out of scope:
 The prior draft was directionally correct but not implementation-safe against the current architecture.
 
 1. **Replay design ignored existing idempotency ledger semantics**
+
 - Problem: timeline design treated every call as `ok/error` and did not model `operation_id` replay/mismatch outcomes.
 - Impact: operators cannot distinguish a real execution from a deterministic replay path.
 - Correction: timeline entries must include `idempotency_status` (`new|replay|mismatch|none`) and optionally `request_hash`.
 
 2. **Tool and CLI wiring was incomplete**
+
 - Problem: draft listed service files only, but current runtime requires full wiring through constants, kernel handlers, tool metadata, catalog, RBAC, docs generation, CLI enum/parser/help/bootstrap.
 - Impact: contract drift and startup/runtime failures.
 - Correction: include full tool/CLI lifecycle wiring as normative work.
 
 3. **Evidence chain placement would break `evidence.latest`**
+
 - Problem: `GateService.evidenceLatest()` currently scans all `*.json` files by mtime in evidence dir.
 - Impact: adding `chain.json` or `manifest.json` can cause `evidence.latest` to return chain metadata instead of latest gate evidence.
 - Correction: constrain `evidenceLatest` to `gate-*.json` and `latest-*.json` only.
 
 4. **Evidence chain algorithm was insufficiently deterministic**
+
 - Problem: hash contract depended on ad hoc concatenation with timestamps and no algorithm versioning/canonical object format.
 - Impact: brittle verification and future incompatibility.
 - Correction: define canonical hash payload with explicit `algorithm_version` and stable field ordering.
 
 5. **Concurrency guarantees were underspecified**
+
 - Problem: replay sequence allocation and chain append were specified without lock strategy.
 - Impact: sequence collisions/interleaving corruption under concurrent tool calls and parallel gates.
 - Correction: require per-run replay lock and per-feature evidence-chain lock using existing file-lock primitives.
 
 6. **Doctor checks were too vague for deterministic implementation**
+
 - Problem: checks were listed by category but not by stable check IDs, severity rules, or deterministic ordering.
 - Impact: non-repeatable output and fragile tests.
 - Correction: define fixed check IDs, output ordering, and strict severity semantics.
 
 7. **Feature gating model was missing from policy schema**
+
 - Problem: draft said “feature-flagged where needed” but provided no schema/default additions.
 - Impact: policy validation failure if teams add unmodeled fields.
 - Correction: add policy/default/schema contracts for observability and diagnostics toggles.
 
 8. **Rollout and retention interactions were not addressed**
+
 - Problem: no plan for replay file growth, no compatibility plan with evidence retention/pruning.
 - Impact: unbounded artifacts or false integrity failures.
 - Correction: add retention policy and explicit integrity-vs-pruning rules.
@@ -139,7 +151,7 @@ innovation:
 observability:
   replay:
     enabled: true
-    write_mode: best_effort     # best_effort | required
+    write_mode: best_effort # best_effort | required
     retention:
       max_runs: 100
       max_entries_per_run: 50000
@@ -151,10 +163,11 @@ observability:
 diagnostics:
   doctor:
     enabled: true
-    strict_fails_on: critical   # critical | warning
+    strict_fails_on: critical # critical | warning
 ```
 
 Normative behavior:
+
 - `innovation.replay_debugger` and `observability.replay.enabled` must both be true for timeline writes.
 - `innovation.tamper_evident_evidence` and `observability.evidence_integrity.enabled` must both be true for chain writes.
 - `write_mode=best_effort` logs failures but never fails tool execution; `required` fails the parent tool call.
@@ -191,6 +204,7 @@ Entry contract (`timeline.jsonl` line):
 ```
 
 Rules:
+
 - `sequence` strictly increases per `run_id`.
 - write path is append-only JSONL; no in-place mutation of prior lines.
 - `idempotency_status` must be one of `none|new|replay|mismatch`.
@@ -234,6 +248,7 @@ Hash payload (canonical JSON, stable key order):
 ```
 
 Rules:
+
 - `chain_hash = sha256(canonical_json(hash_payload))`.
 - `manifest.integrity.json` stores `entry_count`, `head_hash`, `last_verified_at`, `last_verified_result`.
 - verification fails closed on missing artifact, hash mismatch, or chain discontinuity.
@@ -272,6 +287,7 @@ Report contract:
 ```
 
 Rules:
+
 - findings sorted lexically by `id`.
 - summary counts derived strictly from findings array.
 - `ok = (critical===0)` in normal mode.
@@ -281,12 +297,15 @@ Rules:
 ### New Tools
 
 1. `replay.timeline_get` (read-only)
+
 - query timeline by `run_id` with optional `from_sequence`, `limit`, `feature_id`, `tool_name` filters.
 
 2. `evidence.verify_chain` (read-only)
+
 - verify integrity chain for a feature and return validity summary + first failure details.
 
 3. `doctor.run` (mutating)
+
 - execute deterministic checks and persist doctor report artifacts.
 
 ### Required Wiring for Every New Tool
@@ -303,11 +322,13 @@ Rules:
 ## 3.6 CLI Surface Changes
 
 Add commands:
+
 - `aop replay --run-id <id> [--from-sequence N] [--limit N] [--feature-id <id>] [--tool <name>] [--json]`
 - `aop evidence verify --feature-id <id> [--json]`
 - `aop doctor [--strict] [--json]`
 
 Required CLI wiring:
+
 - `apps/control-plane/src/cli/types.ts` (`CliCommand` additions + option fields)
 - `apps/control-plane/src/cli/cli-argument-parser.ts`
 - `apps/control-plane/src/cli/help-command-handler.ts`
@@ -317,6 +338,7 @@ Required CLI wiring:
 ## 3.7 Error Code Additions
 
 Add to `apps/control-plane/src/core/error-codes.ts`:
+
 - `replay_run_not_found`
 - `replay_timeline_corrupt`
 - `replay_timeline_write_failed`
@@ -335,9 +357,11 @@ Add to `apps/control-plane/src/core/error-codes.ts`:
 ## OID0: Contract Scaffolding and Policy Wiring
 
 ### Goals
+
 - land schema/default/tool scaffolding first to prevent runtime/config drift
 
 ### File Targets
+
 - `agentic/orchestrator/defaults/policy.defaults.yaml`
 - `agentic/orchestrator/schemas/policy.schema.json`
 - `agentic/orchestrator/schemas/policy.user.schema.json`
@@ -354,10 +378,12 @@ Add to `apps/control-plane/src/core/error-codes.ts`:
 - `agentic/orchestrator/tools.md`
 
 ### Mandatory Checks
+
 - `npm run validate:mcp-contracts`
 - `npm run validate:architecture`
 
 Exit criteria:
+
 - policy composition validates with new fields
 - no contract validator drift
 
@@ -398,6 +424,7 @@ Instrument `ToolRuntime` finalization path with deterministic timeline writes us
 - extend `apps/control-plane/test/mcp.spec.ts` for `replay.timeline_get` parity
 
 Exit criteria:
+
 - identical tool call sequence produces byte-identical timeline lines (except timestamps)
 - sequence strictly monotonic under concurrent writes
 
@@ -412,6 +439,7 @@ Add chain writer/validator service and integrate into gate evidence write path a
 ### Critical Integration Correction
 
 Before chain integration, change `GateService.evidenceLatest()` selection logic:
+
 - include only files matching `gate-*.json` and `latest-*.json`
 - exclude `chain.jsonl`, `manifest.integrity.json`, and any non-gate evidence metadata
 
@@ -450,6 +478,7 @@ Before chain integration, change `GateService.evidenceLatest()` selection logic:
 - extend `apps/control-plane/test/mcp.spec.ts` for `evidence.verify_chain` parity
 
 Exit criteria:
+
 - tampered artifact triggers deterministic failure reason
 - untampered chain verifies successfully
 
@@ -497,6 +526,7 @@ Implement deterministic scanner with fixed check IDs and explicit severity mappi
 - extend `apps/control-plane/test/mcp.spec.ts` for `doctor.run` parity
 
 Exit criteria:
+
 - report artifact is schema-valid
 - strict mode exit behavior is deterministic and tested
 
@@ -505,6 +535,7 @@ Exit criteria:
 ## OID4: Full Tool + CLI Surface Integration
 
 ### Goals
+
 - complete all runtime and CLI wiring for new commands/tools
 
 ### Tool Integration Targets
@@ -532,6 +563,7 @@ Exit criteria:
 - `npm run typecheck`
 
 Exit criteria:
+
 - tools callable via both in-process and MCP clients
 - CLI commands documented and discoverable via `aop help`
 
@@ -542,6 +574,7 @@ Exit criteria:
 ### Replay Retention
 
 Implement deterministic cleanup:
+
 - keep newest `max_runs` timeline files
 - cap each run by `max_entries_per_run` with rollover file strategy or truncation policy (must be deterministic and documented)
 
@@ -549,6 +582,7 @@ Implement deterministic cleanup:
 
 If a separate evidence pruning policy exists, integrity-tracked files must not be pruned silently.
 Required v1 behavior:
+
 - doctor emits `DRIFT_007_EVIDENCE_CHAIN_INVALID` if chain references missing files.
 - documentation explicitly states integrity verification requires preserving tracked artifacts.
 
@@ -564,6 +598,7 @@ Required v1 behavior:
 - doctor drift test for missing chained artifact
 
 Exit criteria:
+
 - replay growth bounded by policy
 - integrity failures are diagnosable, not silent
 
@@ -636,18 +671,22 @@ Exit criteria:
 ## 7. Core Tradeoffs
 
 1. **Replay coverage vs overhead**
+
 - More per-call fields increase diagnostic value but add write overhead.
 - Chosen tradeoff: lightweight JSONL entries with optional `request_hash`, default best-effort writes.
 
 2. **Integrity strictness vs operational flexibility**
+
 - Fail-closed verification improves audit trust but can break workflows under artifact pruning.
 - Chosen tradeoff: integrity is opt-in, with explicit doctor findings for retention drift.
 
 3. **Doctor breadth vs determinism**
+
 - Broad checks can produce noisy, environment-dependent output.
 - Chosen tradeoff: fixed local checks with stable IDs/severities, no external network probes.
 
 4. **Command ergonomics vs parser simplicity**
+
 - Nested command (`aop evidence verify`) is user-friendly but requires explicit parser subcommand logic.
 - Chosen tradeoff: implement subcommand support explicitly rather than introducing ambiguous top-level aliases.
 
@@ -672,6 +711,7 @@ Exit criteria:
 ## 9. Definition of Done
 
 A completed implementation session MUST provide:
+
 - changed files and rationale by phase (OID0-OID6)
 - schema/catalog/RBAC deltas and regenerated `tools.md`
 - test additions and executed command results