agentic-orchestrator 0.1.27 → 0.2.0

package/docs/superpowers/specs/2026-03-29-organizer-sidecar-runtime-loop-design.md

@@ -0,0 +1,225 @@
+# Organizer Sidecar Runtime Loop — Design Spec
+
+**Date:** 2026-03-29
+**Status:** Draft
+**Scope:** Layer 1 "brain" — the runtime loop that invokes the LLM to analyze specs and write the ordering artifact to disk.
+
+## 1. Context
+
+The ordering agent has two layers:
+
+- **Layer 2 (already implemented):** `OrganizerAgentEnrollmentScheduler` — a stateless adapter that reads a pre-computed `organizer-ordering.json` artifact and maps entries to `EnrollmentDecision[]`. Falls back to `CapacityAwareEnrollmentScheduler` when the artifact is missing, corrupt, partial, or expired.
+- **Layer 1 (this spec):** `OrganizerSidecarService` — a polling-based service that collects pending feature specs, sends them to the organizer LLM session, parses/validates the structured response, and writes the ordering artifact to disk.
+
+The sidecar follows the established reconciler global-session pattern: one persistent LLM session per run, created/reattached by `SessionOrchestrator`. Unlike the reconciler (which uses `sendMessage()` to dispatch fire-and-forget work), the organizer uses `provider.runWorker()` because it needs the LLM's text response back to parse the structured JSON ordering result.
+
+## 2. Component: OrganizerSidecarService
+
+**Location:** `apps/control-plane/src/supervisor/organizer-sidecar-service.ts`
+
+### 2.1 Responsibilities
+
+1. Poll on a configurable interval (default 10s) to detect backlog changes.
+2. Collect spec paths from execution control (pending requests) and runtime state (active/queued features).
+3. Read spec file contents from disk.
+4. Build a prompt message containing all spec contents.
+5. Invoke `provider.runWorker()` with the organizer role and the spec content as `instructions`.
+6. Extract the text response from `WorkerRunOutput`, parse as JSON, validate against the AJV schema.
+7. On validation failure: retry once with a correction prompt.
+8. On success: write the artifact atomically (temp + rename).
+9. On persistent failure: log warning, leave existing artifact in place.
+
+### 2.2 Dependencies (injected)
+
+| Dependency            | Type                                | Purpose                                        |
+| :-------------------- | :---------------------------------- | :--------------------------------------------- |
+| `provider`            | `WorkerProvider`                    | `runWorker()` to invoke organizer LLM          |
+| `organizerSessionId`  | `() => string \| null`              | Getter for current organizer session ID        |
+| `kernel`              | `SupervisorKernelPort`              | Read execution control, feature state          |
+| `state`               | `SupervisorRuntimeState`            | `runId`, `ownerInstanceId`, active feature IDs |
+| `artifactPath`        | `string`                            | Output path for `organizer-ordering.json`      |
+| `schemaValidator`     | `ValidateFunction`                  | AJV compiled validator                         |
+| `pollIntervalMs`      | `number` (optional, default 10000)  | Polling interval                               |
+| `maxSpecContentBytes` | `number` (optional, default 102400) | Size guard for spec content                    |
+
+## 3. Polling Loop & Change Detection
+
+### 3.1 Loop mechanics
+
+```
+start() -> setInterval(pollIntervalMs) -> each tick calls reconcile()
+stop()  -> clearInterval + await in-flight reconcile
+```
+
+Follows the `ExecutionEnrollmentService` pattern exactly. Timer is `unref()`'d to avoid blocking Node.js exit.
+
+### 3.2 Concurrency guard
+
+If a reconcile is already in-flight, the next tick awaits it rather than starting a parallel one. Same serialization pattern as `ExecutionEnrollmentService.reconcile()`.
+
+### 3.3 Change detection
+
+Each tick:
+
+1. Read all pending execution requests from execution control.
+2. Read active feature IDs from `state.sessionsByFeature` and queued IDs from `state.queue`.
+3. Compute a fingerprint via `stableHash()` of: `{pending request IDs, feature IDs, spec_paths}`.
+4. Compare to `lastFingerprint`. If unchanged, skip the LLM call.
+5. If changed, proceed with spec collection and LLM dispatch.
+
+The fingerprint also becomes the `backlog_fingerprint` field in the written artifact.
+
+## 4. Spec Collection & Prompt Construction
+
+### 4.1 Spec collection
+
+1. **Execution control:** Read `execution-control.json`, extract `features[].spec_path` from all pending requests.
+2. **Runtime state:** Get active feature IDs from `state.sessionsByFeature` and queued IDs from `state.queue`. Read each feature's state frontmatter for `spec_path`.
+3. **Deduplicate** by feature ID (a feature may appear in both sources).
+4. **Read spec files** from disk. Skip any that don't exist (log warning).
+
+### 4.2 Prompt message format
+
+```
+Analyze the following feature specs and produce an ordering artifact.
+
+Active instance: <instanceId>
+Total features: <N>
+
+--- FEATURE: <feature_id_1> ---
+spec_path: <path>
+status: <current status or "pending">
+
+<full spec file content>
+
+--- FEATURE: <feature_id_2> ---
+spec_path: <path>
+status: <current status or "pending">
+
+<full spec file content>
+
+---
+
+Produce a single JSON object with the ordering analysis. No markdown fences.
+```
+
+### 4.3 Size guard
+
+If total spec content exceeds `maxSpecContentBytes` (default 100KB), include only the first N specs that fit. Log a warning. The resulting artifact will have `completion_status: 'partial'`.
+
+## 5. System Prompt Enhancement
+
+The existing `config/agentic/orchestrator/prompts/organizer.system.md` will be enhanced with a **Dependency Detection Signals** section inserted between "Analysis discipline" and "Dependency classification":
+
+### 5.1 Explicit signals (highest confidence)
+
+- `Depends On:` field in spec frontmatter or body — authoritative, never ignore.
+- Spec A references spec B by feature ID (e.g. "requires feature-xyz to be merged first").
+- Spec B references spec A — if spec B says "builds on feature-xyz" or "extends the work in feature-xyz," that means B depends on A.
+- Spec explicitly lists file paths that another spec also explicitly lists — when both specs declare they will modify the same file(s), the one that creates/restructures goes first.
+- "After X is complete" / "Once X lands" language in either direction.
+
+### 5.2 Structural signals (medium confidence -> `inferred_dependency`)
+
+- Two specs modify the same file paths — the one that creates/restructures should go first.
+- Spec A adds an interface that spec B consumes (import/type dependency).
+- Spec A creates a database table/schema that spec B queries.
+- Spec A introduces a configuration key that spec B reads.
+
+### 5.3 Semantic signals (lower confidence -- classify carefully)
+
+- Specs in the same domain area (e.g. both touch auth) don't necessarily depend on each other — only flag if one's changes would conflict with or build upon the other's.
+- Shared terminology alone is not a dependency — look for actual data/control flow relationships.
+- If two specs could be implemented in either order without merge conflicts or behavioral issues, they are independent.
+
+### 5.4 Not dependencies
+
+- Specs that happen to be in the same directory but touch different concerns.
+- Specs that share a programming language or framework.
+- Specs that were submitted in the same request (ordering within a request is the organizer's job to determine, not assume).
+
+## 6. Response Parsing & Validation
+
+### 6.1 Parse flow
+
+1. Extract text from `WorkerRunOutput` (provider-specific key, typically `text` or `response`). Try `JSON.parse()` on the full text. If that fails, extract substring from first `{` to last `}` and parse that.
+2. Unwrap the `{ type: "ORDERING", ordering: { ... } }` envelope to get the inner object.
+3. Augment with metadata fields the LLM doesn't produce:
+   - `version: 1`
+   - `generated_at: nowIso()`
+   - `expires_at`: current time + configurable TTL (default 120s)
+   - `completion_status: 'complete'` (or `'partial'` if size guard triggered)
+   - `backlog_fingerprint`: the fingerprint from change detection
+   - `analyzed_requests`: the request IDs included in this analysis
+4. Validate the assembled artifact against `organizer-ordering-artifact.schema.json` using AJV.
+
+### 6.2 Retry on validation failure
+
+- Call `runWorker()` again with a correction instruction: `"Your previous response was invalid: <AJV error summary>. Please produce the ordering JSON again with the same specs."`
+- Parse and validate the retry response identically.
+- If retry also fails: log warning, do not write artifact. Enrollment scheduler falls back to capacity-aware.
+
+### 6.3 Atomic write on success
+
+- Write to temp file: `organizer-ordering.json.tmp`
+- Rename to final path: `organizer-ordering.json`
+- Atomic on POSIX (same-directory rename).
+- Update `lastFingerprint` so next tick skips unless backlog changes again.
+
+## 7. Wiring & Lifecycle
+
+### 7.1 runtime.ts changes
+
+When `organizerEnabled` is true:
+
+1. Add `'organizer'` case to `resolveRoleSessionId()` returning `this.state.organizerSessionId`.
+2. Instantiate `OrganizerSidecarService` with provider, session ID getter, kernel, state, artifact path, and AJV validator.
+3. After `sessionOrchestrator.ensureGlobalOrchestratorSession()` completes (which creates the organizer session), call `organizerSidecar.start()`.
+4. On run teardown, call `organizerSidecar.stop()` before closing sessions.
+
+### 7.2 When organizer is not configured
+
+- No `OrganizerSidecarService` is created.
+- `OrganizerAgentEnrollmentScheduler` is not used (already handled).
+- Zero behavioral change from current state.
+
+## 8. File Map
+
+| File                                                      | Action | Description                              |
+| :-------------------------------------------------------- | :----- | :--------------------------------------- |
+| `supervisor/organizer-sidecar-service.ts`                 | Create | The service class                        |
+| `supervisor/runtime.ts`                                   | Modify | Instantiate and wire the service         |
+| `config/agentic/orchestrator/prompts/organizer.system.md` | Modify | Add dependency detection signals section |
+| `test/organizer-sidecar-service.spec.ts`                  | Create | Unit tests                               |
+| `test/supervisor.unit.spec.ts`                            | Modify | Update runtime wiring tests if needed    |
+
+**No changes to:** `organizer-enrollment-scheduler.ts`, `session-orchestrator.ts`, `run-coordinator.ts`, `execution-enrollment-service.ts`.
+
+## 9. Testing Strategy
+
+### 9.1 Unit tests (organizer-sidecar-service.spec.ts)
+
+1. GIVEN backlog unchanged WHEN tick fires THEN no sendMessage call.
+2. GIVEN new pending request WHEN tick fires THEN sends spec content to organizer session.
+3. GIVEN valid LLM response WHEN parsed THEN writes artifact atomically.
+4. GIVEN malformed LLM response WHEN first attempt fails THEN retries with correction prompt.
+5. GIVEN malformed response on retry WHEN second attempt fails THEN logs warning and skips write.
+6. GIVEN no organizer session ID WHEN tick fires THEN skips gracefully.
+7. GIVEN spec file missing from disk WHEN collecting specs THEN skips that feature with warning.
+8. GIVEN total spec content exceeds size limit WHEN building prompt THEN produces partial artifact.
+9. GIVEN service started WHEN stop() called THEN awaits in-flight reconcile before resolving.
+10. GIVEN valid artifact WHEN schema validation runs THEN passes AJV check.
+11. GIVEN artifact with invalid classification enum WHEN schema validation runs THEN rejects.
+
+### 9.2 Mocking approach
+
+- Stub `provider.runWorker()` to return canned `WorkerRunOutput` with text responses.
+- Stub kernel methods for execution control and feature state reads.
+- Use `makeTempRepo` for artifact file writes.
+
+## 10. Out of Scope
+
+- Changes to the `OrganizerAgentEnrollmentScheduler` (Layer 2 adapter).
+- Changes to `SessionOrchestrator` (organizer session creation already implemented).
+- Human-in-the-loop escalation for `ambiguous_dependency_requires_human` classifications (future work).
+- Multi-provider organizer support (uses the same provider as the run).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentic-orchestrator",
-  "version": "0.1.27",
+  "version": "0.2.0",
   "private": false,
   "workspaces": [
     "packages/*"
@@ -11,6 +11,7 @@
     "nx": "node scripts/nx-safe.mjs",
     "test": "npm run nx -- run control-plane:test",
     "run": "tsx apps/control-plane/src/cli/aop.ts",
+    "run:debug": "node --inspect dist/apps/control-plane/cli/aop.js",
     "build": "npm run lint && npm run nx -- run control-plane:build",
     "verify": "npm run typecheck && npm run validate:all && npm run lint:fix && npm run format && npm run test && npm run nx -- run control-plane:build && npm run dashboard:build",
     "typecheck": "tsc -p tsconfig.json --noEmit",
@@ -50,6 +51,6 @@
     "ajv": "^8.18.0",
     "chalk": "^5.6.2",
     "minimatch": "^10.2.4",
-    "yaml": "^2.8.2"
+    "yaml": "2.8.3"
   }
 }

package/packages/web-dashboard/package.json

@@ -7,7 +7,8 @@
     "dev": "next dev",
     "build": "next build",
     "start": "next start",
-    "typecheck": "tsc -p tsconfig.json --noEmit"
+    "typecheck": "tsc -p tsconfig.json --noEmit",
+    "test": "vitest run --config vitest.config.ts"
   },
   "dependencies": {
     "@monaco-editor/react": "^4.7.0",

package/packages/web-dashboard/src/app/analytics/page.tsx

@@ -11,8 +11,13 @@ import type {
   ProviderAnalytics,
 } from '@/lib/types.js';
 
+interface RoleProviderAnalytics extends ProviderAnalytics {
+  role: string;
+}
+
 interface AnalyticsPayload {
   providers: ProviderAnalytics[];
+  by_role?: RoleProviderAnalytics[];
   recent_outcomes: Array<{
     feature_id: string;
     status: string;
@@ -21,6 +26,8 @@ interface AnalyticsPayload {
     duration_ms: number;
     cost_usd: number;
     recorded_at: string;
+    role?: string;
+    tokens_used?: number;
   }>;
   interactive_metrics: InteractivePerformanceMetrics;
 }
@@ -100,6 +107,7 @@ export default function AnalyticsPage() {
   const [histogram, setHistogram] = useState<number[]>([]);
   const [sortColumn, setSortColumn] = useState<SortColumn>('success_rate');
   const [sortDirection, setSortDirection] = useState<SortDirection>('desc');
+  const [roleFilter, setRoleFilter] = useState<string>('all');
 
   useEffect(() => {
     const params = new URLSearchParams(window.location.search);
@@ -166,7 +174,11 @@ export default function AnalyticsPage() {
   }, [project]);
 
   const sortedProviders = useMemo(() => {
-    const copy = [...analytics.providers];
+    const source =
+      roleFilter !== 'all' && analytics.by_role
+        ? analytics.by_role.filter((entry) => entry.role === roleFilter)
+        : analytics.providers;
+    const copy = [...source];
     copy.sort((left, right) => {
       const direction = sortDirection === 'asc' ? 1 : -1;
       if (sortColumn === 'provider' || sortColumn === 'model') {
@@ -175,7 +187,15 @@ export default function AnalyticsPage() {
       return (Number(left[sortColumn]) - Number(right[sortColumn])) * direction;
     });
     return copy;
-  }, [analytics.providers, sortColumn, sortDirection]);
+  }, [analytics.providers, analytics.by_role, roleFilter, sortColumn, sortDirection]);
+
+  const availableRoles = useMemo(() => {
+    const roles = new Set<string>();
+    for (const entry of analytics.by_role ?? []) {
+      roles.add(entry.role);
+    }
+    return ['all', ...Array.from(roles).sort((a, b) => a.localeCompare(b))];
+  }, [analytics.by_role]);
 
   const updateSort = (column: SortColumn) => {
     if (sortColumn === column) {
@@ -337,6 +357,20 @@ export default function AnalyticsPage() {
 
       <section className={styles.section}>
         <h2 className={styles.sectionTitle}>Provider Performance</h2>
+        {availableRoles.length > 1 ? (
+          <div className={styles.badgeRow}>
+            {availableRoles.map((role) => (
+              <button
+                key={role}
+                type="button"
+                className={`${styles.badge} ${role === roleFilter ? styles.badgePass : styles.badgeNeutral}`}
+                onClick={() => setRoleFilter(role)}
+              >
+                {role === 'all' ? 'All Roles' : role}
+              </button>
+            ))}
+          </div>
+        ) : null}
         <div className={styles.tableWrapper}>
           <table className={styles.table}>
             <thead>