agentic-orchestrator 0.1.28 → 0.2.0

package/spec-files/outstanding/agentic_orchestrator_phase_specific_agent_profiles_and_token_telemetry_spec.md

@@ -0,0 +1,303 @@
+# Feature Spec: Phase-Specific Agent Profiles and Token Telemetry (AOP)
+
+> **Purpose of this document**: Define per-phase provider/model selection for planner, builder, and QA, and add trustworthy token telemetry so operators can measure the cost impact of those choices.
+
+**Version:** 1.0
+**Date:** 2026-03-16
+**Status:** Draft
+**Roadmap Mapping:** M53
+**Depends On:**
+
+- `spec-files/outstanding/agentic_orchestrator_dynamic_agent_swapping_spec.md`
+- `spec-files/completed/agentic_orchestrator_reactive_execution_enrollment_spec.md`
+- `spec-files/completed/agentic_orchestrator_dashboard_advanced_ux_spec.md`
+
+---
+
+## 0. Scope
+
+This spec covers the following user requests:
+
+1. Support different agent/model selections per phase or role:
+   - planner -> Agent/Model A
+   - builder -> Agent/Model B
+   - qa -> Agent/Model C
+2. Provide the telemetry required to make those cost-saving choices observable.
+3. Replace the current “token usage” placeholder behavior with real usage reporting when available and explicit `unavailable` reporting when not.
+
+Out of scope:
+
+- automatic blocked-state escalation and agent swapping policy logic
+- dashboard route redesign
+- policy-file splitting
+
+### 0.1 Required Standards
+
+All implementation derived from this spec MUST preserve:
+
+- deterministic provider/session resolution per worker role
+- normalized error handling for unsupported providers, missing credentials, and incompatible execution modes
+- truthful usage reporting with explicit unavailable states where telemetry is not exposed
+- compatibility with the existing cost, analytics, and reporting artifacts unless intentionally versioned
+
+### 0.2 OOP and Architecture Guardrails (Mandatory)
+
+Implementation MUST follow:
+
+- **KISS**: treat per-role selection and token telemetry as two focused seams, not one catch-all “optimization” subsystem.
+- **SRP**: provider resolution, usage capture, persistence, and dashboard presentation MUST remain separate responsibilities.
+- **SOLID-oriented design**:
+  - open/closed for future roles/providers/usage fields without branching through monolithic handlers
+  - interface segregation between provider-selection contracts and usage-reporting contracts
+  - dependency inversion so reporting consumes normalized usage data instead of provider-specific payloads
+- **Hexagonal alignment**: provider-specific parsing stays in provider/runtime adapters; application services consume normalized metadata only.
+- **Composition over inheritance**: layer role-specific selection and usage instrumentation through composed helpers and services.
+- **High cohesion / low coupling**: token telemetry changes must not be tightly coupled to dashboard-only rendering concerns.
+
+---
+
+## 1. Problem Statement
+
+The codebase already contains partial role-specific selection infrastructure:
+
+- `resolveProviderSelectionsByRole(...)`
+- `runtime.role_provider_overrides` in `agents.yaml`
+
+But the current CLI contract still forces all per-role overrides to use the same provider as the base selection, which blocks the intended cost-optimization use case.
+
+At the same time, token usage surfaces exist in reporting and dashboard UI, but no provider path is reliably recording real usage into `cost.record`. That makes the “Feature Usage” UI mostly decorative rather than operational.
+
+The result is a half-finished optimization story:
+
+- configuration hints exist
+- cross-phase agent selection is not actually allowed end-to-end
+- usage metrics do not prove whether the configuration is helping
+
+---
+
+## 2. Design Summary
+
+### 2.1 Phase-Specific Profiles
+
+Make per-role runtime selection a first-class supported feature.
+
+Terminology:
+
+- `role` means the runtime worker role: `planner`, `builder`, `qa`, and optionally `reconciler`
+- `phase-specific` in this spec maps to those role selections because the orchestrator executes the phases through those role lanes
+
+### 2.2 Supported Overrides
+
+`agents.yaml` MUST support role-specific overrides for:
+
+- `provider`
+- `model`
+- `provider_config_env`
+- `agent_config`
+
+The base runtime defaults remain the fallback.
+
+### 2.3 Token Telemetry Truthfulness
+
+Token telemetry MUST be truthful rather than inferred silently.
+
+Normative rule:
+
+- if the provider exposes token usage, record it
+- if the provider does not expose token usage, mark usage as `unavailable`
+- do not fabricate zeros for missing provider telemetry
+
+Derived USD remains optional and secondary.
+
+---
+
+## 3. Functional Requirements
+
+### FR-1 Allow Cross-Provider Role Overrides
+
+The runtime MUST allow role-specific selections whose provider differs from the base runtime provider.
+
+Acceptance criteria:
+
+1. `planner`, `builder`, and `qa` can each resolve to a different provider/model pair.
+2. CLI validation no longer rejects role overrides merely because the provider differs from the base selection.
+3. Runtime session metadata persists the actual provider/model used for each worker lane.
+
+### FR-2 Preserve Safe Selection Rules
+
+Per-role selection MUST still preserve existing safety and validation rules.
+
+Acceptance criteria:
+
+1. Unsupported providers still fail through `ERROR_CODES.UNSUPPORTED_AGENT_PROVIDER`.
+2. Missing credentials still fail per provider.
+3. Execution mode compatibility is validated per selected provider.
+
+### FR-3 Expose Role Usage in Runtime Artifacts
+
+Each worker execution or session lane MUST persist enough metadata to answer:
+
+- which provider/model handled planner?
+- which provider/model handled builder?
+- which provider/model handled qa?
+- how many tokens were consumed by each lane?
+
+Acceptance criteria:
+
+1. Worker event journal entries capture provider/model and usage metadata when available.
+2. Dashboard analytics can aggregate token usage by provider/model and by role.
+3. Feature-level summaries can show planner/builder/qa usage breakdowns when available.
+
+### FR-4 Real Token Recording
+
+Implement actual token usage recording into the existing cost pipeline.
+
+Recommended data shape:
+
+```json
+{
+  "feature_id": "feature_checkout_flow",
+  "tokens_used": 31244,
+  "estimated_cost_usd": 0.62,
+  "recorded_at": "2026-03-16T12:00:00.000Z",
+  "by_role": {
+    "planner": { "tokens_used": 4200, "status": "reported" },
+    "builder": { "tokens_used": 22100, "status": "reported" },
+    "qa": { "tokens_used": 4944, "status": "reported" }
+  }
+}
+```
+
+Acceptance criteria:
+
+1. Provider adapters or decision-loop integration call `cost.record` with real deltas when the provider returns usage.
+2. Missing provider usage is recorded as `status: unavailable`, not `0`.
+3. Existing budget and dashboard reporting surfaces consume the new real values.
+
+### FR-5 Dashboard and Analytics Updates
+
+The dashboard MUST present token usage in a way that supports model-cost decisions.
+
+Acceptance criteria:
+
+1. Feature detail shows total tokens plus role-level breakdown when available.
+2. Analytics route can compare providers/models by token usage, estimated cost, and success metrics.
+3. UI states clearly distinguish `reported`, `estimated`, and `unavailable`.
+
+---
+
+## 4. Configuration Contract
+
+### 4.1 `agents.yaml`
+
+The existing `runtime.role_provider_overrides` field becomes normative rather than experimental.
+
+Example:
+
+```yaml
+runtime:
+  default_provider: codex
+  default_model: gpt-5.2-codex
+  role_provider_overrides:
+    planner:
+      provider: claude
+      model: claude-sonnet-4
+    builder:
+      provider: codex
+      model: gpt-5.2-codex
+    qa:
+      provider: openai
+      model: gpt-5.2
+```
+
+### 4.2 Compatibility with Dynamic Agent Swapping
+
+This spec is complementary to `agentic_orchestrator_dynamic_agent_swapping_spec.md`.
+
+Boundary:
+
+- this spec defines the baseline per-role selection
+- the dynamic-agent-swapping spec defines post-block escalation to alternate profiles
+
+Dynamic swapping MUST treat the role-specific baseline from this spec as the starting configuration.
+
+---
+
+## 5. Implementation Notes
+
+### 5.1 Existing Constraint to Remove
+
+Current `run-command-handler.ts` and `resume-command-handler.ts` reject role-provider overrides whose provider differs from the base provider. That guard must be replaced with compatibility validation, not provider equality validation.
+
+### 5.2 Usage Instrumentation Points
+
+Preferred instrumentation order:
+
+1. provider adapter returns usage in structured form
+2. `WorkerDecisionLoop` or runtime adapter converts it into a normalized usage record
+3. runtime records `cost.record`
+4. reporting/dashboard surfaces read the recorded artifact
+
+### 5.3 CLI Provider Reality
+
+Some CLI providers will not expose token usage.
+
+For those providers, the correct v1 behavior is:
+
+- show `usage unavailable`
+- do not block execution
+- do not silently imply zero cost
+
+---
+
+## 6. File-Level Targets
+
+- `apps/control-plane/src/providers/providers.ts`
+  - allow cross-provider per-role selection and normalize role metadata.
+- `apps/control-plane/src/cli/run-command-handler.ts`
+  - remove same-provider restriction and pass through per-role selections.
+- `apps/control-plane/src/cli/resume-command-handler.ts`
+  - same as run path.
+- `apps/control-plane/src/supervisor/runtime.ts`
+  - persist per-role provider/model metadata into run metadata and feature summaries.
+- `apps/control-plane/src/supervisor/worker-decision-loop.ts`
+  - record per-turn or per-step usage data when available.
+- `apps/control-plane/src/providers/cli-worker-provider.ts`
+  - expose usage metadata if the underlying CLI supports it.
+- `apps/control-plane/src/providers/api-worker-provider.ts`
+  - expose reported usage metadata from API responses.
+- `apps/control-plane/src/application/services/cost-tracking-service.ts`
+  - support richer role-aware usage structure without regressing total summaries.
+- `apps/control-plane/src/application/services/reporting-service.ts`
+  - surface role/provider usage in dashboard/report payloads.
+- `apps/control-plane/src/application/services/performance-analytics-service.ts`
+  - aggregate provider/model token and cost metrics.
+- `packages/web-dashboard/src/components/feature-cost-panel.tsx`
+  - render real usage status and per-role breakdown.
+- `packages/web-dashboard/src/components/dashboard/triage-drawer.tsx`
+  - stop implying token data exists when it does not.
+- `packages/web-dashboard/src/app/analytics/page.tsx`
+  - add usage-driven comparison surfaces.
+- `packages/web-dashboard/src/lib/types.ts`
+  - extend cost/usage types for availability and per-role breakdown.
+
+---
+
+## 7. Acceptance Criteria
+
+1. Planner, builder, and QA can run on different providers/models under one execution.
+2. The runtime persists the actual provider/model used per role.
+3. Token usage is recorded when the provider supplies it.
+4. Missing token data is surfaced explicitly as unavailable rather than as fake zero usage.
+5. Dashboard and analytics surfaces show enough usage detail to evaluate cost-saving model choices.
+
+---
+
+## 8. Risks and Mitigations
+
+- Risk: cross-provider overrides break assumptions in attach/send flows.
+  - Mitigation: keep feature session metadata authoritative per role and validate all role selections at session-creation time.
+- Risk: token telemetry differs across providers.
+  - Mitigation: introduce a normalized usage contract with explicit availability status instead of forcing all providers into the same exact payload.
+- Risk: dashboard cost widgets overstate precision.
+  - Mitigation: always label `reported`, `estimated`, or `unavailable`.

package/spec-files/outstanding/agentic_orchestrator_planning_review_quality_spec.md

@@ -2,9 +2,9 @@
 
 > **Purpose of this document**: Provide implementation-ready delivery for Q1, Q2, Q7, Q8, and Q9 with an explicit architectural critique of the prior draft and a corrected execution plan aligned to the current control-plane architecture.
 
-**Version:** 2.0  
-**Date:** 2026-03-04  
-**Status:** Draft (Architecture-Corrected)  
+**Version:** 2.1  
+**Date:** 2026-03-15  
+**Status:** Draft (Audited Against Current Repo State)  
 **Roadmap Mapping:** M34-M36
 
 ---
@@ -45,7 +45,9 @@ Implementing agents MUST read:
 - `agentic/orchestrator/defaults/policy.defaults.yaml`
 - `packages/web-dashboard/src/lib/aop-client.ts`
 - `packages/web-dashboard/src/lib/types.ts`
-- `packages/web-dashboard/src/app/page.tsx`
+- `packages/web-dashboard/src/components/detail-panel.tsx`
+- `packages/web-dashboard/src/app/analytics/page.tsx`
+- `packages/web-dashboard/src/app/api/features/[id]/triage/route.ts`
 
 ### 0.3 Feature Scope
 
@@ -63,6 +65,16 @@ Out of scope:
 - autonomous prompt rewriting
 - silent gate bypasses without policy and audit metadata
 
+### 0.4 Current Project State
+
+This spec remains largely outstanding as of 2026-03-15, but several adjacent foundations now exist and should not be reimplemented from scratch:
+
+- **Q1 is not implemented.** Intentional collision override support exists, but it is not semantic collision analysis and does not satisfy Q1.
+- **Q2 is not implemented.** The dashboard/status readers can already display `feasibility_score` and `feasibility_breakdown` if those fields exist, but no scoring service or `plan.score` tool exists.
+- **Q7 is not implemented.** There is no prompt-contract harness, fixture set, or CI gate for prompt invariants.
+- **Q8 is not implemented.** The dashboard exposes a thin `/api/flaky` artifact reader and flaky-suspect display, but no flaky-governance service, quarantine mutation tools, or parser-backed rolling statistics pipeline exist.
+- **Q9 is partially prepared but not implemented.** The dashboard already reads `.aop/features/<id>/review_brief.json` and can render a brief if present, but no review-brief generation service or `report.feature_review_brief` tool exists.
+
 ---
 
 ## 1. Architectural Critique of Prior Draft (v1.0)
@@ -622,7 +634,8 @@ Generate deterministic review briefs from canonical runtime artifacts; no free-f
 - `agentic/orchestrator/tools/schemas/output/report.feature_review_brief.output.schema.json` (new)
 - `packages/web-dashboard/src/lib/types.ts`
 - `packages/web-dashboard/src/lib/aop-client.ts`
-- `packages/web-dashboard/src/app/page.tsx`
+- `packages/web-dashboard/src/components/detail-panel.tsx`
+- `packages/web-dashboard/src/app/api/features/[id]/triage/route.ts`
 
 ### Dashboard integration contract
 

package/spec-files/outstanding/agentic_orchestrator_policy_stratification_spec.md

@@ -0,0 +1,225 @@
+# Feature Spec: Internal vs User Policy Stratification (AOP)
+
+> **Purpose of this document**: Define a clearer policy layering model by splitting non-user-editable internal controls from user-editable repository policy overrides.
+
+**Version:** 1.0
+**Date:** 2026-03-16
+**Status:** Draft
+**Roadmap Mapping:** M55
+**Depends On:**
+
+- `spec-files/completed/agentic_orchestrator_init_policy_ux_simplification_spec.md`
+- `spec-files/completed/agentic_orchestrator_validator_hardening_spec.md`
+
+---
+
+## 0. Scope
+
+This spec covers the following user request:
+
+- split policy concerns into `internal.policy.yaml` for non-user-modifiable controls and `policy.yaml` for user-editable controls
+
+Out of scope:
+
+- changing gate semantics
+- replacing `agents.yaml`
+- changing dashboard authoring or runtime standby behavior directly
+
+### 0.1 Required Standards
+
+All implementation derived from this spec MUST preserve:
+
+- schema-driven policy validation
+- deterministic merged-policy composition
+- backward-compatible migration behavior for repos that only have `policy.yaml`
+- normalized startup failures when policy layering is invalid
+
+### 0.2 OOP and Architecture Guardrails (Mandatory)
+
+Implementation MUST follow:
+
+- **KISS**: extend the existing loader into a clear three-layer model instead of introducing multiple overlapping loaders.
+- **SRP**: policy loading, policy validation, init-file generation, and documentation ownership MUST remain separate responsibilities.
+- **SOLID-oriented design**:
+  - open/closed for future protected-path groups or additional policy layers without rewriting callers
+  - interface segregation between full merged-policy validation and user-policy validation
+  - dependency inversion so runtime configuration services depend on a composed-policy contract, not raw file-order assumptions
+- **Hexagonal alignment**: policy composition stays in application/configuration services, not scattered across CLI/runtime consumers.
+- **Composition over inheritance**: compose layered loaders/validators with reusable merge helpers instead of branching subclasses.
+- **High cohesion / low coupling**: internal-policy concerns must not leak into unrelated agent/runtime/dashboard configuration paths.
+
+---
+
+## 1. Problem Statement
+
+The current policy model has two layers:
+
+- bundled `agentic/orchestrator/defaults/policy.defaults.yaml`
+- repo `config/agentic/orchestrator/policy.yaml`
+
+That helps with defaults, but it does not clearly separate:
+
+- controls that ordinary repository users should be able to tune
+- controls that should be fixed or centrally governed
+
+The result is conceptual leakage:
+
+- `policy.defaults.yaml` is package/distribution-level, not repository governance-level
+- `policy.yaml` mixes user-facing knobs with settings that should be treated as protected internal controls
+
+---
+
+## 2. Design Summary
+
+### 2.1 Three-Layer Effective Policy
+
+Effective policy becomes:
+
+1. bundled defaults: `agentic/orchestrator/defaults/policy.defaults.yaml`
+2. repo internal policy: `config/agentic/orchestrator/internal.policy.yaml`
+3. repo user policy: `config/agentic/orchestrator/policy.yaml`
+
+Merge order:
+
+- defaults <- internal <- user
+
+But user overrides MUST be rejected for paths classified as internal-only.
+
+### 2.2 Intent of Each Layer
+
+`policy.defaults.yaml`
+
+- distribution defaults shipped with AOP
+- not the user contract
+
+`internal.policy.yaml`
+
+- repository or platform owner controls
+- safety rails and non-negotiable runtime controls
+- not intended for normal end-user modification
+
+`policy.yaml`
+
+- user-editable operational preferences
+- constrained by user-policy schema
+- cannot override internal-only paths
+
+### 2.3 Loader Behavior
+
+Policy loading MUST:
+
+1. read bundled defaults
+2. read `internal.policy.yaml`
+3. read `policy.yaml`
+4. validate effective merged policy against full schema
+5. validate user policy against a user-facing schema that excludes internal-only fields
+6. fail clearly if `policy.yaml` attempts to override an internal-only path
+
+---
+
+## 3. Functional Requirements
+
+### FR-1 Add `internal.policy.yaml`
+
+Add a first-class internal policy file under the existing config root.
+
+Acceptance criteria:
+
+1. The loader recognizes `config/agentic/orchestrator/internal.policy.yaml`.
+2. Missing internal policy is allowed only if an explicit fallback policy is defined; otherwise init should generate it.
+3. Runtime artifact/debug output can distinguish which layer contributed a final value.
+
+### FR-2 Separate Schemas
+
+Validation must distinguish internal policy from user policy.
+
+Acceptance criteria:
+
+1. Full effective policy still validates against `policy.schema.json`.
+2. User-editable `policy.yaml` validates against a constrained user schema.
+3. Internal-only paths can exist in `internal.policy.yaml` even if absent from user schema.
+
+### FR-3 Reject Forbidden User Overrides
+
+User policy MUST NOT override internal-only paths.
+
+Acceptance criteria:
+
+1. Loader throws a descriptive error when `policy.yaml` attempts to set an internal-only field.
+2. The error includes the blocked path list.
+3. Validation happens before the runtime starts executing features.
+
+### FR-4 Init and Documentation Updates
+
+`aop init` MUST generate the new split configuration intentionally.
+
+Acceptance criteria:
+
+1. Init produces both `internal.policy.yaml` and `policy.yaml`.
+2. Generated comments explain ownership of each file.
+3. AGENTS/CLAUDE/init help/docs reflect the new policy layout.
+
+---
+
+## 4. Internal-Only Examples
+
+The exact list is repository-defined, but initial internal-only candidates SHOULD include controls such as:
+
+- RBAC
+- protected areas
+- mutation idempotency and tool authorization safety rails
+- run-lease ownership and recovery constraints
+- provider execution fail-safe defaults
+- internal observability retention controls where tamper resistance matters
+
+User-editable examples SHOULD include:
+
+- dashboard port and UX toggles
+- notification preferences
+- budget thresholds explicitly intended for operator tuning
+- cleanup grace periods where safe
+
+This split must be encoded as data, not only documentation.
+
+---
+
+## 5. File-Level Targets
+
+- `apps/control-plane/src/application/services/policy-loader-service.ts`
+  - load and merge the new three-layer policy model.
+- `apps/control-plane/src/application/configuration-service.ts`
+  - resolve internal/user policy paths and error messages.
+- `apps/control-plane/src/cli/init-command-handler.ts`
+  - generate and explain both policy files.
+- `agentic/orchestrator/schemas/policy.schema.json`
+  - remain the effective merged-policy schema.
+- `agentic/orchestrator/schemas/policy.user.schema.json`
+  - align to the new user-editable subset.
+- `config/agentic/orchestrator/internal.policy.yaml`
+  - new repo-level internal policy artifact.
+- `config/agentic/orchestrator/policy.yaml`
+  - reduced to user-editable concerns.
+- `AGENTS.md`
+  - update policy guidance.
+- `CLAUDE.md`
+  - keep mirrored guidance aligned.
+
+---
+
+## 6. Acceptance Criteria
+
+1. Effective policy is composed from bundled defaults, internal policy, and user policy.
+2. Internal-only controls live in `internal.policy.yaml`.
+3. User policy cannot override internal-only paths.
+4. `aop init` and documentation present the new split clearly.
+
+---
+
+## 7. Risks and Mitigations
+
+- Risk: migration breaks existing repositories that only have `policy.yaml`.
+  - Mitigation: support a compatibility migration path and clear init/upgrade docs.
+- Risk: ownership boundaries remain implicit.
+  - Mitigation: encode protected paths in schema/loader logic rather than comments alone.
+- Risk: defaults/internal layering becomes harder to debug.
+  - Mitigation: add traceable loader diagnostics describing the final source of each policy path.