agentic-orchestrator 0.1.28 → 0.2.1

package/spec-files/outstanding/agentic_orchestrator_spec_ordering_agent_spec.md

@@ -0,0 +1,370 @@
+# Feature Spec: Spec Ordering Agent for Dependency-Aware Execution (AOP)
+
+> **Purpose of this document**: Define a run-level ordering agent that consumes pending enrollment requests, determines dependency-aware execution order, and feeds the existing enrollment pipeline without becoming part of the planner/builder/QA feature loop.
+
+**Version:** 1.2
+**Date:** 2026-03-28
+**Status:** Draft
+**Roadmap Mapping:** M54
+**Depends On:**
+
+- `spec-files/completed/agentic_orchestrator_reactive_execution_enrollment_spec.md`
+- `spec-files/completed/agentic_orchestrator_add_feature_to_active_execution_spec.md`
+- `spec-files/completed/git-reconciliation-engine.md`
+
+---
+
+## 0. Scope
+
+This spec covers the following user request:
+
+- Add a spec-ordering agent outside the Planning/Building/QA loop that reads waiting specs, determines dependencies and concurrency, and queues execution order accordingly.
+
+Out of scope:
+
+- editing feature specs automatically
+- replacing the planner for per-feature implementation planning
+- replacing lock/collision detection
+- changing merge governance
+- automatic spec file discovery or scanning (specs enter via `aop run`/`aop add` CLI commands)
+
+### 0.1 Required Standards
+
+All implementation derived from this spec MUST preserve:
+
+- single active run ownership and existing enrollment durability guarantees
+- deterministic queue mutation after organizer outputs are normalized
+- auditable organizer decisions and dependency classifications
+- compatibility with the existing enrollment scheduler contract rather than introducing a parallel admission mechanism
+
+### 0.2 OOP and Architecture Guardrails (Mandatory)
+
+Implementation MUST follow:
+
+- **KISS**: introduce one organizer sidecar and one organizer-backed scheduler seam rather than a second orchestration loop.
+- **SRP**: dependency inference, organizer session lifecycle, normalized scheduling decisions, and queue admission MUST remain separate responsibilities.
+- **SOLID-oriented design**:
+  - open/closed for future scheduler strategies or richer dependency evidence without rewriting coordinator logic
+  - interface segregation between organizer inference outputs and enrollment application contracts
+  - dependency inversion so the coordinator depends on scheduler contracts, not organizer implementation details
+- **Hexagonal alignment**: organizer logic belongs beside supervisor/runtime services, not inside feature planners or dashboard routes.
+- **Composition over inheritance**: compose organizer behavior into the enrollment/scheduling system rather than subclassing the coordinator.
+- **High cohesion / low coupling**: organizer decisions must complement, not replace, existing collision, dependency, and merge services.
+
+---
+
+## 1. Problem Statement
+
+The repository already has the key seam for this feature:
+
+- the reactive enrollment spec explicitly defines a pluggable `EnrollmentScheduler` interface (`types.ts:230`)
+- the current implementation ships only the deterministic `CapacityAwareEnrollmentScheduler` (`execution-enrollment-service.ts:128`)
+- the reactive enrollment spec describes `OrganizerAgentEnrollmentScheduler` as the future implementation that plugs into this interface (class name introduced in §3.1; §7.2 discusses organizer-agent compatibility)
+
+That scheduler handles capacity and duplicate rejection, but it does not perform semantic ordering across a backlog of not-yet-executing specs.
+
+The requested capability is a run-level organizer analogous to the existing global reconciler pattern:
+
+- it exists outside feature clusters (like the reconciler managed by `session-orchestrator.ts`)
+- it does not implement the feature itself
+- it decides what should start now vs later based on dependency context
+
+---
+
+## 2. Design Summary
+
+### 2.1 Two-Layer Architecture
+
+The organizer consists of two distinct layers that MUST remain separate:
+
+**Layer 1 — Organizer Sidecar Session (LLM inference)**
+
+A persistent global session (like the reconciler) that:
+
+1. reads spec file contents from `spec_path` on pending `ExecutionRequestFeature` records
+2. derives dependency edges across the full pending backlog
+3. persists a structured ordering artifact to a runtime file (`.aop/runtime/<instanceId>/organizer-ordering.json`)
+4. re-runs analysis when triggered by backlog changes
+
+**Layer 2 — `OrganizerAgentEnrollmentScheduler` (deterministic adapter)**
+
+A stateless scheduler that implements `EnrollmentScheduler.schedule(input)` by:
+
+1. reading the persisted ordering artifact
+2. translating per-request decisions into `EnrollmentSchedulingResult { decisions: EnrollmentDecision[] }`
+3. falling back to `CapacityAwareEnrollmentScheduler` behavior when no ordering artifact exists or for features not covered by the artifact
+
+This separation ensures the LLM inference is decoupled from the deterministic per-request scheduling contract.
+
+### 2.2 Input Universe
+
+The organizer consumes features from pending `ExecutionRequestItem` records in `.aop/runtime/<instanceId>/execution-control.json` — the same pipeline used by `aop run` and `aop add`.
+
+Each `ExecutionRequestItem` contains `features: ExecutionRequestFeature[]` where each feature has a `feature_id` and `spec_path`. The organizer reads spec file contents from those paths.
+
+Additionally, the organizer considers:
+
+- features already active in the current run (to detect cross-feature dependencies)
+- features already queued but not yet started (to avoid redundant ordering)
+- features in terminal enrollment states (`merged`, `failed`, `paused_budget`, `ready_to_merge`) that may satisfy dependency prerequisites. Note: `ready_to_merge` is terminal for enrollment purposes because the feature is no longer consuming an active build pipeline slot.
+
+The organizer does NOT independently discover or scan for spec files. All specs enter through the CLI enrollment commands.
+
+### 2.3 Ordering Strategy
+
+The organizer sidecar session produces a structured ordering artifact containing:
+
+- dependency graph (edges between feature IDs)
+- ready set (features with no unresolved dependencies — all features in the ready set are independent and can be admitted concurrently up to `max_active_features`)
+- deferred set (features blocked by dependencies, with reasons)
+
+Priority of dependency evidence:
+
+1. explicit structured metadata in the spec frontmatter (e.g. `Depends On:` field)
+2. explicit `Depends On:` references in the markdown body
+3. deterministic repository signals: shared file paths, overlapping areas from existing collision detection
+4. semantic inference by the organizer prompt as a fallback, marked as inferred
+
+The organizer MUST classify each dependency edge as one of:
+
+- `explicit_dependency` — declared in spec metadata or body
+- `inferred_dependency` — derived from semantic analysis, lower confidence
+- `no_dependency_found` — specs analyzed and found independent
+- `ambiguous_dependency_requires_human` — confidence too low to decide, escalated
+
+### 2.4 Relationship to Enrollment Scheduler
+
+This spec implements the `OrganizerAgentEnrollmentScheduler` described in the reactive enrollment spec (class name introduced in §3.1; §7.2 discusses organizer-agent compatibility).
+
+Boundary:
+
+- organizer sidecar computes and persists ordering intent (Layer 1)
+- `OrganizerAgentEnrollmentScheduler.schedule()` reads that artifact and produces per-request `EnrollmentDecision[]` (Layer 2)
+- `ExecutionEnrollmentService` and `RunCoordinator` still own durable request application and queue mutation — unchanged
+
+---
+
+## 3. Functional Requirements
+
+### FR-1 Organizer Lives Outside the Feature Loop
+
+The organizer MUST be a global run-level service, not a planner substitute.
+
+Acceptance criteria:
+
+1. It has its own prompt/config/session lifecycle managed by `SessionOrchestrator` (same pattern as the reconciler).
+2. It does not create plan submissions or patches for a feature.
+3. It can continue operating while no feature is actively in planning/building/QA.
+
+### FR-2 Determine Dependency-Aware Ready Set
+
+Given multiple pending enrollment requests, the organizer MUST determine which can run now versus which depend on others.
+
+Acceptance criteria:
+
+1. Specs with no unresolved dependencies can be grouped for concurrent admission.
+2. Specs with unresolved dependencies are deferred with recorded reasons.
+3. Cycles are detected and surfaced as explicit blocking conditions.
+
+### FR-3 Preserve Deterministic Admission
+
+The organizer may use an LLM for semantic reasoning, but final queue mutation remains deterministic and auditable.
+
+Acceptance criteria:
+
+1. The organizer sidecar persists its ordering artifact to disk before any scheduling decisions are made.
+2. The `OrganizerAgentEnrollmentScheduler` reads only the persisted artifact — it never invokes the LLM directly.
+3. Admission order uses deterministic tie-breakers (alphabetical `feature_id`) when multiple specs are equally ready.
+4. Every organizer decision is persisted in the ordering artifact with timestamps and dependency classifications.
+
+### FR-4 Human Escalation for Ambiguity
+
+The organizer MUST not silently invent hard dependencies when confidence is low.
+
+Acceptance criteria:
+
+1. Low-confidence cases are marked `ambiguous_dependency_requires_human`.
+2. Those specs receive `disposition: 'defer'` with reason `dependency_blocked` and a human-readable explanation.
+3. Dashboard/status surfaces can show why the spec is waiting.
+
+### FR-5 Graceful Degradation
+
+The organizer MUST NOT block enrollment when it is unavailable or has not yet analyzed the backlog.
+
+Acceptance criteria:
+
+1. When no ordering artifact exists, `OrganizerAgentEnrollmentScheduler` delegates to `CapacityAwareEnrollmentScheduler` behavior.
+2. When a feature is not covered by the current artifact, it is scheduled using capacity-only logic.
+3. Organizer session failures do not prevent feature enrollment from proceeding.
+
+---
+
+## 4. Contract Additions
+
+### 4.1 Organizer Ordering Artifact
+
+Persisted at `.aop/runtime/<instanceId>/organizer-ordering.json`.
+
+The organizer sidecar MUST write the artifact atomically (write to a temporary file, then rename). The adapter MUST treat a missing or unparseable artifact as absent, triggering fallback behavior (FR-5).
+
+```json
+{
+  "version": 1,
+  "generated_at": "2026-03-16T12:00:00.000Z",
+  "expires_at": "2026-03-16T12:01:00.000Z",
+  "completion_status": "complete",
+  "backlog_fingerprint": "sha256:abc123...",
+  "analyzed_requests": ["request_id_1", "request_id_2"],
+  "dependency_graph": {
+    "spec_1": { "depends_on": [], "classification": "no_dependency_found" },
+    "spec_2": { "depends_on": [], "classification": "no_dependency_found" },
+    "spec_3": { "depends_on": ["spec_2"], "classification": "explicit_dependency" },
+    "spec_4": { "depends_on": ["spec_1"], "classification": "ambiguous_dependency_requires_human" },
+    "spec_5": { "depends_on": [], "classification": "no_dependency_found" }
+  },
+  "ready_set": ["spec_1", "spec_2", "spec_5"],
+  "deferred_set": [
+    {
+      "feature_id": "spec_3",
+      "blocked_by": ["spec_2"],
+      "reason": "explicit dependency: spec references spec_2 in Depends On"
+    }
+  ],
+  "blocked_set": [
+    {
+      "feature_id": "spec_4",
+      "reason": "ambiguous_dependency_requires_human: possible overlap with spec_1 auth module"
+    }
+  ]
+}
+```
+
+### 4.2 Mapping to EnrollmentDecision
+
+The ordering artifact is computed in batch by the sidecar (Layer 1) across the full pending backlog. The adapter (Layer 2) reads this pre-computed artifact during each per-request `schedule()` invocation and looks up the feature being scheduled. This design intentionally separates batch LLM inference from per-request deterministic lookup. The adapter never needs cross-request visibility — the sidecar has already resolved cross-request dependencies into the artifact.
+
+The `OrganizerAgentEnrollmentScheduler.schedule()` translates the ordering artifact into `EnrollmentDecision[]` per the existing contract:
+
+| Artifact classification                          | EnrollmentDecision disposition                 | EnrollmentDecision reason |
+| :----------------------------------------------- | :--------------------------------------------- | :------------------------ |
+| Feature in `ready_set`                           | `queue_now`                                    | `organizer_ready`         |
+| Feature in `deferred_set`                        | `defer`                                        | `dependency_blocked`      |
+| Feature in `blocked_set`                         | `defer`                                        | `dependency_blocked`      |
+| Feature not in artifact                          | Delegate to `CapacityAwareEnrollmentScheduler` | (capacity-based reason)   |
+| Artifact missing, unparseable, partial, or stale | Delegate to `CapacityAwareEnrollmentScheduler` | (capacity-based reason)   |
+
+The scheduler MUST also perform capacity checks (active count vs `max_active_features`) before returning `queue_now` — a feature in the ready set that exceeds capacity receives `defer` with reason `capacity_full`.
+
+### 4.3 Prompt/Session Topology
+
+Add:
+
+- `config/agentic/orchestrator/prompts/organizer.system.md`
+- `organizer` entry in `agents.yaml` under `provider_overrides` (same pattern as `reconciler`)
+
+The organizer follows the same global-sidecar pattern as the reconciler in `session-orchestrator.ts`:
+
+- one session per run, created via `ensureGlobalOrganizerSession()`
+- durable reattach if possible
+- best-effort cleanup on shutdown
+- session ID persisted in `RuntimeSessionsSnapshot`
+
+### 4.4 Scheduler Interface
+
+The `OrganizerAgentEnrollmentScheduler` MUST implement the existing `EnrollmentScheduler` interface:
+
+```typescript
+export class OrganizerAgentEnrollmentScheduler implements EnrollmentScheduler {
+  constructor(deps: {
+    orderingArtifactPath: string;
+    fallback: CapacityAwareEnrollmentScheduler;
+    maxArtifactAgeSecs?: number; // default 60
+  });
+
+  schedule(input: EnrollmentSchedulingInput): Promise<EnrollmentSchedulingResult>;
+}
+```
+
+This is a stateless adapter. It reads the ordering artifact from disk on each `schedule()` call. The organizer sidecar session is responsible for keeping that artifact up to date.
+
+The adapter MUST fall back to the `fallback` scheduler when any of these conditions hold:
+
+1. The artifact file is missing.
+2. The artifact fails JSON parsing.
+3. `completion_status` is `partial`.
+4. `expires_at` is in the past.
+5. The feature is not present in the artifact's `dependency_graph`.
+
+In all fallback cases, the adapter SHOULD log a warning with the fallback reason so operators can diagnose organizer availability issues.
+
+### 4.5 Organizer Trigger Points
+
+The organizer sidecar re-runs analysis when:
+
+1. A new `ExecutionRequestItem` is detected (via `notifyNewRequestHint()` propagation)
+2. A feature reaches a terminal enrollment state (`merged`, `failed`, `paused_budget`) that may unblock deferred specs. The `RunCoordinator` MUST call `executionEnrollmentService.notifyNewRequestHint()` when a feature transitions to a terminal enrollment status. This is in addition to existing notifier events. Without this trigger, deferred features whose blocking dependencies have been satisfied will not be re-evaluated until the next polling interval.
+3. The run coordinator signals a safe checkpoint (existing `noteSafeCheckpoint()` path)
+4. The organizer artifact's `expires_at` has passed, causing the adapter to fall back to capacity-aware scheduling. The enrollment service's existing 2s polling interval serves as the backstop re-evaluation mechanism for deferred features — when the sidecar produces an updated artifact reflecting resolved dependencies, the next poll cycle will read it.
+
+The trigger mechanism uses the same notification pattern as the enrollment service poller — not a separate event bus.
+
+---
+
+## 5. File-Level Targets
+
+- `apps/control-plane/src/supervisor/execution-enrollment-service.ts`
+  - inject `OrganizerAgentEnrollmentScheduler` as the scheduler when organizer is enabled.
+- `apps/control-plane/src/supervisor/run-coordinator.ts`
+  - propagate backlog-change signals to the organizer sidecar session.
+  - forward terminal-state transitions as organizer re-analysis triggers.
+- `apps/control-plane/src/supervisor/session-orchestrator.ts`
+  - add `ensureGlobalOrganizerSession()` following the reconciler pattern.
+  - persist `organizer_session_id` in `RuntimeSessionsSnapshot`.
+- `apps/control-plane/src/supervisor/types.ts`
+  - add `OrganizerOrderingArtifact` type for the persisted ordering artifact.
+  - add `organizer_session_id` to `RuntimeSessionsSnapshot`.
+  - add `organizerSessionId?: string | null` to `SupervisorRuntimeState`.
+  - add `organizer?: string | null` to `PromptBundle`.
+- `apps/control-plane/src/supervisor/prompt-bundle-loader.ts`
+  - add `'organizer'` to the role iteration list (currently `['planner', 'builder', 'qa', 'reconciler']`).
+- `apps/control-plane/src/supervisor/runtime.ts`
+  - wire organizer session creation and `OrganizerAgentEnrollmentScheduler` into the supervisor startup sequence (replacing `CapacityAwareEnrollmentScheduler` when organizer is enabled).
+- `config/agentic/orchestrator/agents.yaml`
+  - add organizer role under `provider_overrides` (same pattern as reconciler).
+- `config/agentic/orchestrator/prompts/organizer.system.md`
+  - new global organizer prompt with spec-analysis instructions.
+- `agentic/orchestrator/schemas/agents.schema.json`
+  - add `organizer` to the `provider_overrides` role enum.
+- `agentic/orchestrator/schemas/organizer-ordering-artifact.schema.json`
+  - JSON schema for the ordering artifact; validated on sidecar write and adapter read.
+- `packages/web-dashboard/src/lib/types.ts`
+  - add organizer ordering artifact type projections for dashboard display (ready/deferred/blocked sets).
+- `packages/web-dashboard/src/app/api/status/route.ts`
+  - expose organizer ready/deferred/blocked summaries from the ordering artifact.
+
+---
+
+## 6. Acceptance Criteria
+
+1. AOP can inspect pending enrollment requests and derive a dependency-aware execution order.
+2. Independent specs can be admitted concurrently up to `max_active_features`.
+3. Deferred specs record the dependencies that kept them out of the ready set in the ordering artifact.
+4. Ambiguous ordering cases are surfaced for human intervention instead of being silently misordered.
+5. The organizer remains outside the planner/builder/QA per-feature loop and feeds the existing enrollment mechanism.
+6. When the organizer is unavailable or has not analyzed a feature, enrollment falls back to capacity-aware scheduling without blocking.
+7. The ordering artifact is a durable, inspectable file that can be read by dashboard and CLI status commands.
+
+---
+
+## 7. Risks and Mitigations
+
+- Risk: semantic dependency inference introduces nondeterministic queue order.
+  - Mitigation: persist normalized ordering artifact to disk; the enrollment scheduler adapter reads only the artifact and applies deterministic tie-breakers. LLM inference is isolated in the sidecar session.
+- Risk: organizer duplicates functionality already present in dependency scheduling.
+  - Mitigation: keep the organizer focused on backlog/spec ordering while preserving existing collision detection and merge governance at feature runtime.
+- Risk: new global sidecar role complicates run topology.
+  - Mitigation: follow the established reconciler sidecar pattern in `session-orchestrator.ts` — one session per run, same lifecycle management.
+- Risk: organizer unavailability blocks all enrollment.
+  - Mitigation: `OrganizerAgentEnrollmentScheduler` falls back to `CapacityAwareEnrollmentScheduler` for unanalyzed features and when no ordering artifact exists (FR-5).
+- Risk: per-request `EnrollmentScheduler.schedule()` interface misaligns with batch backlog analysis.
+  - Mitigation: two-layer design separates batch LLM inference (sidecar → artifact) from per-request scheduling (adapter reads artifact). The adapter is stateless; the sidecar maintains global context.

package/spec-files/outstanding/shadow_workspace_implementation_spec.md

@@ -1242,7 +1242,7 @@ async streamShadowChanges(featureId: string): AsyncIterableIterator<FileChange>
 ### 12.1 Related Specifications
 
 - **Parent Spec:** [Execution Mode Specification](../completed/agentic_orchestrator_execution_mode_spec.md) - Overall architecture for deterministic vs interactive modes
-- **Related:** [Runtime Inspection Specification](./agentic_orchestrator_runtime_inspection_spec.md) - Dashboard integration for shadow monitoring
+- **Related:** [Runtime Inspection Specification](../completed/agentic_orchestrator_runtime_inspection_spec.md) - Dashboard integration for shadow monitoring
 
 ### 12.2 External Resources