agentic-orchestrator 0.1.2 → 0.1.4

package/spec-files/outstanding/agentic_orchestrator_validator_hardening_spec.md

@@ -0,0 +1,365 @@
+# Feature Spec: Validation Script Hardening (AOP)
+
+> **Purpose of this document**: Define implementation-ready improvements to the three existing CI validators (`validate:mcp-contracts`, `validate:docker-mcp`, `validate:architecture`) based on gaps identified through static analysis. Each improvement closes a concrete drift or correctness risk.
+
+**Version:** 1.0  
+**Date:** 2026-03-03  
+**Status:** Draft  
+**Roadmap Mapping:** Infrastructure / Quality
+
+---
+
+## 0. Standards and Dependencies
+
+### 0.1 Required Standards
+
+All implementation MUST preserve:
+- All three validators continue to pass on a clean codebase with zero output on success
+- CI command signatures are unchanged: `npm run validate:mcp-contracts`, `npm run validate:docker-mcp`, `npm run validate:architecture`
+- Exit code 1 on any new failure; exit code 0 only when all checks pass
+- Nx + Vitest coverage thresholds maintained (≥ 90% per file)
+
+### 0.2 Upstream Inputs
+
+Implementing agents MUST read:
+- `agentic/orchestrator/tools/catalog.json` — source of truth for tool metadata
+- `apps/control-plane/src/application/tools/tool-metadata.ts` — `MUTATING_TOOLS` runtime list
+- `apps/control-plane/src/core/constants.ts` — `TOOLS` constants
+- `apps/control-plane/scripts/validate-mcp-contracts.ts` — current MCP validator
+- `apps/control-plane/scripts/validate-architecture-rules.mjs` — current architecture validator
+- `apps/control-plane/scripts/validate-docker-mcp-contract.mjs` — current Docker validator
+- `apps/control-plane/vitest.config.ts` — coverage exclusions
+
+### 0.3 Feature Scope
+
+This spec implements improvements in three areas:
+- **V1** MCP contract cross-validation hardening
+- **V2** Docker MCP contract completeness
+- **V3** Architecture rule completeness and correctness
+
+---
+
+## 1. Objectives
+
+### 1.1 Must-Have Outcomes
+
+- The `mutating` flag in `catalog.json` must be verifiably consistent with `TOOL_BEHAVIOR_METADATA` in code — any skew detected at CI time, not at runtime.
+- All tool input/output schema files referenced in the catalog must be syntactically valid JSON Schema (parseable and compilable by AJV).
+- Schema files present on disk but unreferenced by any catalog entry are reported as orphans.
+- The `supported_roles` field on every catalog tool is validated against the known role set.
+- `application/adapters/` files and `application/multi-project-loader.ts` are subject to defined architecture rules rather than silently skipped.
+- `core` is prohibited from importing `application-services`, `application-tools`, `mcp`, or `providers` layer files.
+- The Docker Dockerfile Node version pin is validated against a minimum major version.
+- The entrypoint passthrough (`exec "$@"`) is verified to be present.
+
+### 1.2 Non-Goals
+
+- No rewrite of validators from scratch — changes are additive to existing checks.
+- No new npm packages beyond `ajv` (already a project dependency).
+- No runtime enforcement changes in kernel or tool runtime (only CI validators change).
+- No coverage gating on the validator scripts themselves (they are scripts, not source).
+
+---
+
+## 2. Architecture Decisions
+
+### 2.1 Validators Remain Scripts
+
+All three validators are plain `.ts` / `.mjs` scripts run via `tsx` or `node`. They do not import from `src/` source — they read files from disk. This remains the pattern.
+
+### 2.2 Catalog Remains the Source of Truth
+
+`catalog.json` is authoritative. The code-side `MUTATING_TOOLS` in `tool-metadata.ts` must match it. When there is a discrepancy, the validator fails and prints the tool name. The developer must reconcile `tool-metadata.ts` to match the catalog.
+
+### 2.3 Architecture Layer Map is Canonical
+
+The layer classification in `validate-architecture-rules.mjs` is the contract. Adding a new source directory without also adding it to the layer map must fail loudly, not silently skip.
+
+---
+
+## 3. Implementation: V1 — MCP Contract Hardening
+
+### 3.1 Cross-Check `mutating` vs `TOOL_BEHAVIOR_METADATA`
+
+**File:** `apps/control-plane/scripts/validate-mcp-contracts.ts`
+
+**Check:** For every tool entry in `catalog.json` with `mutating: true`, verify that the tool's `handler_id` appears as a key in `TOOL_BEHAVIOR_METADATA` (exported from `tool-metadata.ts`) with `mutating: true`.
+
+**Implementation approach:**
+
+The validator is a TypeScript file run via `tsx`. It can import `TOOL_BEHAVIOR_METADATA` directly from the source using a relative path:
+
+```typescript
+import { TOOL_BEHAVIOR_METADATA } from '../src/application/tools/tool-metadata.js';
+```
+
+Then for each tool in the catalog:
+
+```typescript
+const mutatingInCatalog = tool.mutating === true;
+const mutatingInCode = TOOL_BEHAVIOR_METADATA[tool.handler_id]?.mutating === true;
+if (mutatingInCatalog !== mutatingInCode) {
+  errors.push(
+    `Tool "${tool.handler_id}" mutating flag mismatch: catalog=${mutatingInCatalog}, code=${mutatingInCode}`
+  );
+}
+```
+
+**Failure message example:**
+```
+✗ Tool "cost.record" mutating flag mismatch: catalog=true, code=false
+  Fix: add TOOLS.COST_RECORD to MUTATING_TOOLS in tool-metadata.ts
+```
+
+**Current known failure to fix first:** `cost.record` — `TOOLS.COST_RECORD` must be added to `MUTATING_TOOLS` in `tool-metadata.ts` before the check is added, otherwise CI will immediately fail. Fix the code, then add the check.
+
+### 3.2 Validate Input/Output Schemas Are Valid JSON Schema
+
+**File:** `apps/control-plane/scripts/validate-mcp-contracts.ts`
+
+**Check:** For each `input_schema_ref` and `output_schema_ref` in the catalog, load the JSON file and compile it with AJV. A schema that fails to compile (bad `$schema`, invalid keyword, etc.) must be reported.
+
+**Implementation approach:**
+
+```typescript
+import Ajv from 'ajv';
+const ajv = new Ajv({ strict: false });
+
+for (const tool of catalog.tools) {
+  for (const schemaRef of [tool.input_schema_ref, tool.output_schema_ref]) {
+    const schemaPath = path.resolve(toolsDir, schemaRef);
+    const schema = JSON.parse(fs.readFileSync(schemaPath, 'utf8'));
+    try {
+      ajv.compile(schema);
+    } catch (e) {
+      errors.push(`Schema "${schemaRef}" failed AJV compilation: ${e.message}`);
+    }
+  }
+}
+```
+
+`ajv` is already a dependency (`"ajv": "^8.x"` appears in package.json via existing usage in the kernel). No new dependency needed.
+
+### 3.3 Detect Orphan Schema Files
+
+**File:** `apps/control-plane/scripts/validate-mcp-contracts.ts`
+
+**Check:** Collect all `.json` files under `agentic/orchestrator/tools/schemas/input/` and `schemas/output/`. Compare against the set of schema refs mentioned in the catalog. Files on disk that are not referenced by any tool are reported as orphans.
+
+**Known current orphans (informational, not blocking — these may be shared base schemas):**
+- `schemas/input/mutating.schema.json`
+- `schemas/input/read.schema.json`
+- `schemas/output/standard_success.schema.json`
+
+These three files should be either:
+1. Added as `$ref` base schemas used by at least one tool, or
+2. Explicitly listed in a `known_shared_schemas` array in the validator (not subject to orphan check).
+
+**Implementation approach:**
+
+```typescript
+const referencedSchemas = new Set<string>();
+for (const tool of catalog.tools) {
+  referencedSchemas.add(tool.input_schema_ref);
+  referencedSchemas.add(tool.output_schema_ref);
+}
+
+const allSchemaFiles = [
+  ...glob.sync('schemas/input/*.json', { cwd: toolsDir }),
+  ...glob.sync('schemas/output/*.json', { cwd: toolsDir }),
+];
+
+const knownShared = new Set(['schemas/input/mutating.schema.json', 'schemas/input/read.schema.json', 'schemas/output/standard_success.schema.json']);
+
+for (const file of allSchemaFiles) {
+  if (!referencedSchemas.has(file) && !knownShared.has(file)) {
+    warnings.push(`Orphan schema file not referenced by any catalog tool: ${file}`);
+  }
+}
+```
+
+Orphans emit **warnings** (printed but do not fail CI) unless the file is outside the `knownShared` exemption list.
+
+### 3.4 Validate `supported_roles` Against Known Role Set
+
+**File:** `apps/control-plane/scripts/validate-mcp-contracts.ts`
+
+**Check:** Every value in every tool's `supported_roles` array must be one of: `orchestrator`, `planner`, `builder`, `qa`, `system`.
+
+```typescript
+const KNOWN_ROLES = new Set(['orchestrator', 'planner', 'builder', 'qa', 'system']);
+
+for (const tool of catalog.tools) {
+  for (const role of tool.supported_roles) {
+    if (!KNOWN_ROLES.has(role)) {
+      errors.push(`Tool "${tool.name}" has unknown role "${role}" in supported_roles`);
+    }
+  }
+}
+```
+
+---
+
+## 4. Implementation: V2 — Docker MCP Contract Completeness
+
+### 4.1 Validate Node Major Version Pin
+
+**File:** `apps/control-plane/scripts/validate-docker-mcp-contract.mjs`
+
+**Check:** The `FROM` line in the Dockerfile must pin to a specific Node major version (`node:22` or higher). `node:latest`, `node:lts`, or any unversioned form must fail.
+
+```javascript
+const MIN_NODE_MAJOR = 22;
+
+const fromMatch = dockerfile.match(/^FROM\s+node:(\d+)/m);
+assert(fromMatch, 'Dockerfile must use a pinned Node version (e.g. node:22-bookworm-slim)');
+const nodeMajor = parseInt(fromMatch[1], 10);
+assert(
+  nodeMajor >= MIN_NODE_MAJOR,
+  `Dockerfile Node version ${nodeMajor} is below minimum required ${MIN_NODE_MAJOR}`
+);
+```
+
+### 4.2 Verify Entrypoint `exec "$@"` Passthrough
+
+**File:** `apps/control-plane/scripts/validate-docker-mcp-contract.mjs`
+
+**Check:** The entrypoint shell script (referenced in `ENTRYPOINT` of the Dockerfile) must contain `exec "$@"` as the final command, ensuring the container actually runs the requested process after startup checks.
+
+```javascript
+const entrypointPath = path.resolve(repoRoot, 'docker', 'entrypoint.sh');
+const entrypoint = fs.readFileSync(entrypointPath, 'utf8');
+assert(
+  /\bexec\s+"\$@"/.test(entrypoint),
+  'Entrypoint script must contain exec "$@" passthrough'
+);
+```
+
+---
+
+## 5. Implementation: V3 — Architecture Rule Completeness
+
+### 5.1 Classify `application/adapters/` and `application/multi-project-loader.ts`
+
+**File:** `apps/control-plane/scripts/validate-architecture-rules.mjs`
+
+**Problem:** Files in `application/` that are not under `application/services/` or `application/tools/` are currently classified as `'unknown'` and skipped. These files can import from any layer without detection.
+
+**Fix:** Add an `'application'` layer classification and rules. Files that match `application/` but not the more specific `application/services/` or `application/tools/` paths resolve to `'application'`.
+
+**Layer classification update:**
+
+```javascript
+function classifyFile(relativePath) {
+  if (relativePath.startsWith('application/services/')) return 'application-services';
+  if (relativePath.startsWith('application/tools/')) return 'application-tools';
+  if (relativePath.startsWith('application/')) return 'application';  // NEW
+  if (relativePath.startsWith('cli/')) return 'cli';
+  if (relativePath.startsWith('core/')) return 'core';
+  if (relativePath.startsWith('interfaces/')) return 'interfaces';
+  if (relativePath.startsWith('mcp/')) return 'mcp';
+  if (relativePath.startsWith('providers/')) return 'providers';
+  if (relativePath.startsWith('supervisor/')) return 'supervisor';
+  return 'unknown';
+}
+```
+
+**Architecture rules addition:**
+
+```javascript
+const ARCHITECTURE_RULES = {
+  // existing rules ...
+  'application': ['cli', 'interfaces', 'supervisor'],  // NEW: same restrictions as application-services
+};
+```
+
+**Unknown layer handling:** Files that remain `'unknown'` after classification must fail the validator with a message, rather than being silently skipped. This ensures new directories added to `src/` cannot evade architecture rules:
+
+```javascript
+if (layer === 'unknown') {
+  errors.push(`Unclassified source file "${relativePath}" — add it to the layer map in validate-architecture-rules.mjs`);
+  continue;
+}
+```
+
+The top-level `index.ts` (public barrel export) should be explicitly exempt:
+
+```javascript
+const EXEMPT_FILES = new Set(['index.ts']);
+```
+
+### 5.2 Prohibit `core` from Importing Upward Layers
+
+**File:** `apps/control-plane/scripts/validate-architecture-rules.mjs`
+
+**Problem:** The current `core` rule forbids `['cli', 'interfaces', 'supervisor']` but does not forbid `application-services`, `application-tools`, `mcp`, or `providers`. `core` should be a pure domain layer — no upward dependencies.
+
+**Fix:** Expand the `core` forbidden list:
+
+```javascript
+'core': ['cli', 'interfaces', 'supervisor', 'application-services', 'application-tools', 'application', 'mcp', 'providers'],
+```
+
+Before adding this rule, verify that no existing `core/` file currently imports from those layers. If any do, they must be refactored to eliminate the import first.
+
+**Verification command:**
+```bash
+grep -r "from.*application\|from.*mcp\|from.*providers" apps/control-plane/src/core/ --include="*.ts"
+```
+
+---
+
+## 6. Acceptance Criteria
+
+### V1 — MCP Contract Hardening
+
+- [ ] `npm run validate:mcp-contracts` fails if any catalog tool with `mutating: true` is not in `TOOL_BEHAVIOR_METADATA`
+- [ ] `npm run validate:mcp-contracts` fails if any input or output schema fails AJV compilation
+- [ ] `npm run validate:mcp-contracts` prints warnings for orphan schema files not in the `knownShared` list
+- [ ] `npm run validate:mcp-contracts` fails if any `supported_roles` value is not in the known role set
+- [ ] `cost.record` tool has `TOOLS.COST_RECORD` added to `MUTATING_TOOLS` in `tool-metadata.ts`
+- [ ] All existing passing behaviors of `validate:mcp-contracts` continue to pass
+
+### V2 — Docker MCP Contract
+
+- [ ] `npm run validate:docker-mcp` fails if the Dockerfile `FROM` does not pin to `node:22` or higher
+- [ ] `npm run validate:docker-mcp` fails if the entrypoint script does not contain `exec "$@"`
+- [ ] All existing passing behaviors of `validate:docker-mcp` continue to pass
+
+### V3 — Architecture Rules
+
+- [ ] `npm run validate:architecture` fails for any file in `application/adapters/` that imports from `cli`, `interfaces`, or `supervisor`
+- [ ] `npm run validate:architecture` fails for any file in `core/` that imports from `application-services`, `application-tools`, `application`, `mcp`, or `providers`
+- [ ] `npm run validate:architecture` fails for any unclassified source file (not in the exempt list)
+- [ ] `index.ts` is explicitly exempted from architecture classification
+- [ ] All existing passing behaviors of `validate:architecture` continue to pass
+
+---
+
+## 7. Implementation Order
+
+Implement in this order to avoid false CI failures:
+
+1. **Fix `cost.record` in `tool-metadata.ts`** — adds `TOOLS.COST_RECORD` to `MUTATING_TOOLS` (code fix, not validator fix)
+2. **Verify `core/` has no upward imports** — grep check before changing rules
+3. **V3: Classify `application/` layer and `unknown` fail-loudly** — low risk, additive classification
+4. **V3: Expand `core` forbidden list** — only after step 2 confirms no violations exist
+5. **V1: Add `mutating` cross-check** — only after step 1 ensures it passes immediately
+6. **V1: Add AJV schema compilation check** — compile all schemas, fix any that fail
+7. **V1: Add orphan schema detection** — populate `knownShared` with the 3 known orphans
+8. **V1: Add `supported_roles` validation** — straightforward set membership check
+9. **V2: Add Node version pin check** — regex against Dockerfile `FROM` line
+10. **V2: Add `exec "$@"` check** — regex against entrypoint script
+
+---
+
+## 8. Files Affected
+
+| File | Change Type |
+|---|---|
+| `apps/control-plane/scripts/validate-mcp-contracts.ts` | Add checks 3.1–3.4 |
+| `apps/control-plane/scripts/validate-architecture-rules.mjs` | Add checks 5.1–5.2 |
+| `apps/control-plane/scripts/validate-docker-mcp-contract.mjs` | Add checks 4.1–4.2 |
+| `apps/control-plane/src/application/tools/tool-metadata.ts` | Add `TOOLS.COST_RECORD` to `MUTATING_TOOLS` |
+| `apps/control-plane/test/tool-metadata.spec.ts` | Add tests for `COST_RECORD` mutating behavior |
+| `spec-files/progress.md` | Update with completed milestone |