agentic-orchestrator 0.1.4 → 0.1.6

package/spec-files/outstanding/agentic_orchestrator_artifact_database_publishing_spec.md

@@ -0,0 +1,500 @@
+# Feature Spec: `.aop` Artifact Publishing to Configurable Audit Database (AOP)
+
+> **Purpose of this document**: Define implementation-ready functionality to publish runtime artifacts from `.aop/**` to a user-configured database after execution, enabling durable audit history of what the orchestrator did and why it did it. The feature includes external database targets and a built-in local Docker-backed database mode (`type: local`).
+
+**Version:** 1.0  
+**Date:** 2026-03-04  
+**Status:** Draft  
+**Roadmap Mapping:** M40
+
+---
+
+## 0. Standards and Dependencies
+
+### 0.1 Required Standards
+
+All implementation MUST preserve:
+- deterministic runtime behavior and MCP/in-process parity
+- existing command behavior when publishing is disabled
+- normalized error envelope (`{ ok: false, error: { code, message, details } }`)
+- atomic file-write/lock guarantees for `.aop/**` artifacts
+- Nx + Vitest + coverage thresholds (>= 90% per file)
+
+### 0.2 Upstream Inputs
+
+Implementing agents MUST read:
+- `apps/control-plane/src/core/path-layout.ts`
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/supervisor/run-coordinator.ts`
+- `apps/control-plane/src/cli/run-command-handler.ts`
+- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
+- `apps/control-plane/src/cli/cli-argument-parser.ts`
+- `apps/control-plane/src/cli/help-command-handler.ts`
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `apps/control-plane/src/mcp/operation-ledger.ts`
+- `agentic/orchestrator/policy.yaml`
+- `agentic/orchestrator/defaults/policy.defaults.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `agentic/orchestrator/tools/catalog.json`
+- `spec-files/completed/agentic_orchestrator_dot_aop_generated_artifacts_spec.md`
+
+### 0.3 Feature Scope
+
+This spec implements three slices:
+- **DB1** Configurable artifact publishing policy + database connection contract
+- **DB2** Post-run and manual publishing pipeline with idempotent checkpoints
+- **DB3** Local Docker database mode (`database.type: local`)
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- Users can configure a database target in YAML and publish `.aop/**` artifacts to it.
+- Publishing can run automatically after `aop run` execution and can also be triggered manually.
+- Published records are queryable over time by `run_id`, `feature_id`, tool activity, gate results, and decisions.
+- `database.type: local` starts/uses a local Docker database automatically and publishes there without external infra setup.
+- Publishing is idempotent and safe to retry.
+
+### 1.2 Non-Goals
+
+- No replacement of file-backed `.aop/**` runtime storage.
+- No requirement to support every SQL engine in first release.
+- No real-time streaming pipeline in this phase (batch publish at execution boundaries).
+- No dashboard UI implementation in this phase (data plane + CLI/tool contracts only).
+
+---
+
+## 2. Architecture Decisions
+
+### 2.1 Source of Truth Remains `.aop/**`
+
+Database rows are derived artifacts for analytics/audit. `.aop/**` remains authoritative runtime state.
+
+### 2.2 First-Class Publisher Service
+
+All publishing must route through a single service boundary (`ArtifactPublisherService`) to centralize:
+- artifact enumeration
+- normalization and hashing
+- DB transaction semantics
+- checkpointing/retry behavior
+
+### 2.3 Idempotency by `(target_id, run_id, publish_revision)`
+
+Publishing writes must be safely repeatable. Re-running publish for the same run must not duplicate logical records.
+
+### 2.4 Local Mode Resolves to Docker-Managed Postgres
+
+`database.type: local` is a convenience wrapper around a managed local Postgres container. External engines remain explicitly configured.
+
+### 2.5 Failure Behavior is Configurable
+
+Default is best-effort (run succeeds even if publish fails). Strict mode is available when audit persistence is mandatory.
+
+---
+
+## 3. Configuration Contract
+
+### 3.1 Policy Additions (`policy.yaml`)
+
+Add `artifact_publishing` section to policy (full + defaults + schema):
+
+```yaml
+artifact_publishing:
+  enabled: false
+  trigger: post_run           # post_run | manual_only
+  mode: best_effort           # best_effort | required
+  include_content:
+    state_md: true
+    plan_json: true
+    decisions_md: true
+    qa_index_json: true
+    gate_logs_preview_bytes: 16384
+  database:
+    type: postgres            # postgres | local
+    target_id: default
+    postgres:
+      connection_uri: "${AOP_AUDIT_DB_URL}"
+      schema: aop_audit
+      ssl_mode: require       # disable | require
+      connect_timeout_ms: 8000
+    local:
+      engine: postgres
+      image: postgres:16-alpine
+      container_name: aop-audit-db
+      port: 55432
+      database: aop_audit
+      username: aop
+      password_env: AOP_LOCAL_AUDIT_DB_PASSWORD
+      volume_name: aop_audit_db_data
+      auto_start: true
+      wait_timeout_ms: 30000
+```
+
+### 3.2 YAML Validation
+
+`agentic/orchestrator/schemas/policy.schema.json` MUST validate the new section with:
+- strict enums for `trigger`, `mode`, `database.type`
+- required `postgres.connection_uri` when `type: postgres`
+- required local fields when `type: local`
+- numeric bounds for port/timeouts/preview bytes
+
+### 3.3 Secret Handling
+
+- connection strings and passwords MUST support env interpolation (`${ENV_VAR}`).
+- raw password literals in committed policy SHOULD be rejected with explicit validation guidance.
+
+### 3.4 Defaults
+
+`agentic/orchestrator/defaults/policy.defaults.yaml` MUST include `artifact_publishing.enabled: false` so behavior remains opt-in.
+
+---
+
+## 4. Canonical Publish Payload
+
+### 4.1 Artifacts Included Per Run
+
+Minimum dataset for a run publish:
+- `.aop/features/index.json`
+- `.aop/runtime/operation-ledger/<run_id>.json`
+- for each feature in run scope:
+  - `state.md`
+  - `plan.json` (if present)
+  - `qa_test_index.json` (if present)
+  - `cost.json` (if present)
+  - `decisions.md` (if present)
+  - gate/evidence metadata and selected content previews
+
+### 4.2 Normalized Record Categories
+
+Publisher MUST normalize into these logical categories:
+- `run_summary`
+- `feature_snapshot`
+- `tool_execution`
+- `gate_execution`
+- `decision_log_entry`
+- `artifact_manifest_entry`
+- `publish_event`
+
+### 4.3 Hashing and Determinism
+
+For each artifact row:
+- include `content_sha256`
+- include `relative_path`
+- normalize JSON key order before hashing JSON content
+- ensure stable sort order by `feature_id`, then `relative_path`
+
+---
+
+## 5. Database Schema (v1)
+
+### 5.1 Required Tables
+
+1. `aop_publish_runs`
+- `publish_id` (PK)
+- `target_id`
+- `run_id`
+- `publish_revision`
+- `status` (`started|succeeded|failed`)
+- `started_at`, `finished_at`
+- `error_code`, `error_message`
+
+2. `aop_runs`
+- `run_id` (PK)
+- `project_root_hash`
+- `started_at`
+- `finished_at`
+- `queue_depth`
+- `runtime_status`
+
+3. `aop_features`
+- composite PK: (`run_id`, `feature_id`)
+- `status`
+- `branch`
+- `gate_profile`
+- `last_updated`
+- `cost_estimated_usd`
+- `tokens_used`
+
+4. `aop_tool_executions`
+- composite PK: (`run_id`, `operation_id`)
+- `feature_id`
+- `tool_name`
+- `actor_type`
+- `result`
+- `error_code`
+- `latency_ms`
+- `request_hash`
+- `created_at`, `updated_at`
+
+5. `aop_gate_executions`
+- surrogate PK + unique (`run_id`, `feature_id`, `gate_name`, `executed_at`)
+- `result`, `exit_code`
+- `mode`
+- `evidence_path`
+- `logs_preview`
+
+6. `aop_decisions`
+- surrogate PK + index (`run_id`, `feature_id`, `line_no`)
+- `entry_text`
+- `entry_hash`
+
+7. `aop_artifact_manifest`
+- composite PK: (`run_id`, `feature_id`, `relative_path`)
+- `size_bytes`
+- `content_sha256`
+- `content_preview`
+- `modified_at`
+
+### 5.2 Migration Contract
+
+- Create `schema_migrations` table and apply ordered SQL migrations.
+- Publisher must run `ensureSchema()` before first write for a target.
+
+---
+
+## 6. Runtime Flow and Trigger Points
+
+### 6.1 Automatic Post-Run Publishing
+
+`RunCommandHandler.execute(...)` flow extension:
+1. run supervisor cycle (`supervisor.start(...)`)
+2. if `artifact_publishing.enabled && trigger == post_run`, invoke publisher
+3. attach publish result under response payload:
+   - `data.artifact_publish.status`
+   - `data.artifact_publish.publish_id`
+   - `data.artifact_publish.warning` (best-effort failure only)
+
+### 6.2 Manual Publishing
+
+Add CLI command:
+- `aop publish [--run-id <id>] [--latest] [--dry-run] [--strict]`
+
+Behavior:
+- default `--latest` when no run id provided
+- `--dry-run` builds manifest + stats without DB writes
+- `--strict` forces non-zero exit on publish failure regardless of policy mode
+
+### 6.3 MCP Tool Surface
+
+Add tools:
+- `artifacts.publish` (mutating: true)
+- `artifacts.publish_status` (mutating: false)
+
+`artifacts.publish` input includes `run_id`, `dry_run`, `strict`, `trigger_source`.
+
+---
+
+## 7. Local Docker Database Mode (`type: local`)
+
+### 7.1 Behavior
+
+When `database.type: local`:
+1. verify Docker CLI availability
+2. verify/start container using configured image/name/port/volume
+3. wait for DB health/readiness
+4. connect via computed Postgres URI
+5. run schema migrations and publish
+
+### 7.2 Local Runtime Artifacts
+
+Persist local DB runtime metadata under:
+- `.aop/runtime/artifact-publishing/local-db.json`
+
+Contract example:
+
+```json
+{
+  "target_id": "default",
+  "container_name": "aop-audit-db",
+  "engine": "postgres",
+  "host": "127.0.0.1",
+  "port": 55432,
+  "database": "aop_audit",
+  "username": "aop",
+  "started_at": "2026-03-03T22:00:00Z"
+}
+```
+
+### 7.3 Docker Failure Handling
+
+Errors must map cleanly:
+- `artifact_publish_local_docker_unavailable`
+- `artifact_publish_local_start_failed`
+- `artifact_publish_local_unhealthy`
+
+If mode is `best_effort`, run continues and publish failure is reported in payload + logs.
+
+---
+
+## 8. Error Codes
+
+Add to `apps/control-plane/src/core/error-codes.ts`:
+- `artifact_publish_config_invalid`
+- `artifact_publish_target_unconfigured`
+- `artifact_publish_connect_failed`
+- `artifact_publish_migration_failed`
+- `artifact_publish_write_failed`
+- `artifact_publish_checkpoint_conflict`
+- `artifact_publish_local_docker_unavailable`
+- `artifact_publish_local_start_failed`
+- `artifact_publish_local_unhealthy`
+
+---
+
+## 9. Implementation Plan
+
+## 9.1 DB1: Configuration + Validation
+
+### DB1-T1 Policy contract
+- update `policy.yaml` examples
+- update `policy.defaults.yaml`
+- extend `policy.schema.json`
+
+### DB1-T2 Typed policy accessors
+- add typed readers for `artifact_publishing` section in kernel/service layer
+
+### DB1-T3 Config normalization
+- implement env interpolation + normalization helper
+
+## 9.2 DB2: Publisher pipeline
+
+### DB2-T1 Artifact enumeration
+- deterministic scanner for run/feature artifacts
+- manifest builder with hashes and size
+
+### DB2-T2 Store interface and Postgres adapter
+- `ArtifactAuditStore` interface
+- `PostgresArtifactAuditStore` implementation (transactional upsert)
+
+### DB2-T3 Checkpointing and idempotency
+- write local checkpoint under `.aop/runtime/artifact-publishing/checkpoints/<run_id>.json`
+- support safe retries after partial failures
+
+### DB2-T4 Trigger integration
+- invoke publisher from run flow and manual command path
+
+## 9.3 DB3: Local Docker orchestration
+
+### DB3-T1 Local DB manager
+- start/inspect/wait for container
+- compute connection URI for local target
+
+### DB3-T2 Local-only smoke checks
+- optional smoke test command for local mode readiness
+
+---
+
+## 10. File Targets
+
+### 10.1 New files
+- `apps/control-plane/src/application/services/artifact-publisher-service.ts`
+- `apps/control-plane/src/application/services/artifact-manifest-service.ts`
+- `apps/control-plane/src/application/services/artifact-audit-store.ts`
+- `apps/control-plane/src/application/services/postgres-artifact-audit-store.ts`
+- `apps/control-plane/src/application/services/local-artifact-db-manager.ts`
+- `apps/control-plane/src/cli/publish-command-handler.ts`
+- `agentic/orchestrator/tools/schemas/input/artifacts.publish.input.schema.json`
+- `agentic/orchestrator/tools/schemas/output/artifacts.publish.output.schema.json`
+- `agentic/orchestrator/tools/schemas/input/artifacts.publish_status.input.schema.json`
+- `agentic/orchestrator/tools/schemas/output/artifacts.publish_status.output.schema.json`
+- `apps/control-plane/src/application/services/sql/artifact-publishing/postgres/001_init.sql`
+
+### 10.2 Existing files to modify
+- `apps/control-plane/src/core/kernel.ts`
+- `apps/control-plane/src/core/constants.ts`
+- `apps/control-plane/src/core/error-codes.ts`
+- `apps/control-plane/src/core/path-layout.ts`
+- `apps/control-plane/src/cli/cli-argument-parser.ts`
+- `apps/control-plane/src/cli/help-command-handler.ts`
+- `apps/control-plane/src/cli/types.ts`
+- `apps/control-plane/src/interfaces/cli/bootstrap.ts`
+- `apps/control-plane/src/cli/run-command-handler.ts`
+- `agentic/orchestrator/tools/catalog.json`
+- `agentic/orchestrator/tools.md`
+- `agentic/orchestrator/policy.yaml`
+- `agentic/orchestrator/defaults/policy.defaults.yaml`
+- `agentic/orchestrator/schemas/policy.schema.json`
+- `README.md`
+
+### 10.3 Tests
+- `apps/control-plane/test/artifact-publisher.spec.ts` (new)
+- `apps/control-plane/test/local-artifact-db-manager.spec.ts` (new)
+- `apps/control-plane/test/run-command.spec.ts` (extend post-run publish assertions)
+- `apps/control-plane/test/cli.unit.spec.ts` (parse + publish command)
+- `apps/control-plane/test/mcp.spec.ts` (new tool registration + parity)
+- `apps/control-plane/test/kernel.spec.ts` (tool handler contracts)
+
+---
+
+## 11. Acceptance Criteria
+
+### DB1 — Configurable DB contract
+
+- [ ] `artifact_publishing` policy section schema-validates with strict enum/type checks
+- [ ] env-variable interpolation is supported for DB secrets
+- [ ] invalid config surfaces `artifact_publish_config_invalid` with actionable details
+
+### DB2 — Publish pipeline
+
+- [ ] `aop run` publishes artifacts post-run when enabled and trigger is `post_run`
+- [ ] `aop publish --run-id <id>` publishes historical run artifacts on demand
+- [ ] published dataset includes run summary, feature snapshots, tool executions, decisions, and artifact manifest
+- [ ] re-running publish for same run is idempotent (no duplicate logical records)
+- [ ] best-effort mode does not fail run command on publish failure; required mode does
+
+### DB3 — Local Docker mode
+
+- [ ] `database.type: local` auto-starts/uses configured Docker DB container
+- [ ] publish succeeds to local DB without external DB provisioning
+- [ ] explicit, typed errors are returned when Docker is missing/unhealthy
+
+### Quality and regression
+
+- [ ] existing commands continue to work unchanged when publishing is disabled
+- [ ] `npm run typecheck` and `npm test` pass with coverage thresholds preserved
+- [ ] `npm run validate:mcp-contracts` passes with updated tool catalog/schemas
+
+---
+
+## 12. Implementation Order
+
+1. Add policy contract + schema + defaults (`artifact_publishing`, disabled by default)
+2. Add new error codes and constants for publish tools
+3. Implement manifest + hashing service (no DB yet)
+4. Implement Postgres audit store + migrations
+5. Add `artifacts.publish` / `artifacts.publish_status` tool handlers and schemas
+6. Add `aop publish` CLI command and bootstrap wiring
+7. Wire post-run trigger in `RunCommandHandler`
+8. Implement `type: local` Docker manager and integrate with publisher
+9. Add tests (unit + integration contracts)
+10. Update docs and examples
+
+---
+
+## 13. Risks and Mitigations
+
+- Risk: large artifact payloads cause slow publish times.
+  - Mitigation: default to metadata + bounded previews for large logs, plus batched writes.
+- Risk: local Docker startup flakiness creates noisy failures.
+  - Mitigation: explicit readiness checks, bounded retries, and best-effort default mode.
+- Risk: duplicate records on retries.
+  - Mitigation: deterministic upsert keys and run-level publish checkpoints.
+- Risk: secrets leakage in logs.
+  - Mitigation: redact connection URI credentials and password fields from all logs/errors.
+
+---
+
+## 14. Open Questions
+
+1. Should first release support only Postgres (`postgres` + `local`) or include MySQL in MVP?
+2. Should `aop status` expose latest publish status summary by default?
+3. Should strict mode be policy-only, CLI-only, or both (recommended: both)?
+
+---
+
+## 15. Progress Tracking Note
+
+When implementation starts, add milestone entries to `spec-files/progress.md` and move this spec from `spec-files/outstanding/` to `spec-files/completed/` only after acceptance criteria are met.