agentic-orchestrator 0.1.26 → 0.1.28

package/spec-files/outstanding/agentic_orchestrator_runtime_inspection_spec.md

@@ -0,0 +1,940 @@
+# Feature Spec: Runtime File Inspection (AOP Dashboard)
+
+> Purpose: Enable direct inspection of `.aop/runtime` artifacts (Operation Ledger and Worker Events) through the dashboard with time-sorted, formatted views per feature.
+
+**Version:** 1.0
+**Date:** 2026-03-05
+**Status:** Outstanding
+**Roadmap Mapping:** M46
+**Depends On:** M45 (Dashboard Advanced UX)
+
+---
+
+## 0. Scope and Standards
+
+### 0.1 Primary User Job
+
+**Operator/Debugger Job:** Understand what happened during a feature run by inspecting:
+
+- Operation Ledger: idempotent tool invocations with request/response payloads
+- Worker Events: agent lifecycle events (started, completed, failed, timeout)
+
+**Key Questions This Feature Answers:**
+
+1. What tools were called for this feature and what were the results?
+2. When did the agent start/stop/fail/timeout?
+3. What was the sequence of events leading to a failure?
+4. Were there any retries or duplicate operations?
+
+### 0.2 UX Principles
+
+1. **Time-first ordering:** All events sorted by timestamp (newest first by default)
+2. **Progressive disclosure:** Raw JSON collapsed by default, formatted summary visible
+3. **Feature-scoped:** Filter to single feature ID, not global view
+4. **Read-only:** No mutation of runtime files through UI
+5. **Performance-bounded:** Limit display to last N events, paginate if needed
+
+### 0.3 Quality Standards
+
+- TypeScript strict mode passes
+- ESLint zero warnings
+- Coverage >= 90% for new code
+- API response time < 200ms for typical feature (< 100 events)
+- Graceful degradation when files are large (> 1000 events)
+- Must pass `npm run verify` checks
+
+---
+
+## 1. Runtime File Structure
+
+### 1.1 Run ID Tracking in Feature State
+
+**Schema Change Required:** Add `run_history` field to `state.schema.json`
+
+**Location:** `.aop/features/<feature_id>/state.md` (YAML frontmatter)
+
+**New Field:**
+
+```yaml
+run_history:
+  - run_id: 'run:8efe99b2-bba0-473a-ad2e-22c54153fdd8'
+    started_at: '2026-03-05T22:24:39.635Z'
+    ended_at: '2026-03-05T23:00:09.599Z'
+    status: 'failed'
+  - run_id: 'run:69c36d92-3255-435a-806a-6425461e5aa8'
+    started_at: '2026-03-05T06:05:12.123Z'
+    ended_at: '2026-03-05T06:07:17.975Z'
+    status: 'completed'
+```
+
+**Schema Definition:**
+
+```json
+{
+  "run_history": {
+    "type": "array",
+    "description": "Time-sorted list of run IDs associated with this feature (newest first). Updated by supervisor on run start/end.",
+    "items": {
+      "type": "object",
+      "required": ["run_id", "started_at"],
+      "properties": {
+        "run_id": {
+          "type": "string",
+          "description": "Run ID from runtime_sessions"
+        },
+        "started_at": {
+          "type": "string",
+          "format": "date-time",
+          "description": "When this run started working on this feature"
+        },
+        "ended_at": {
+          "type": "string",
+          "format": "date-time",
+          "description": "When this run stopped working on this feature (null if still active)"
+        },
+        "status": {
+          "type": "string",
+          "enum": ["active", "completed", "failed", "timeout"],
+          "description": "Final status of this run for this feature"
+        }
+      }
+    }
+  }
+}
+```
+
+**Update Behavior:**
+
+- **On feature.init:** Create `run_history` with current run_id, status `active`
+- **On supervisor start:** Append current run_id to `run_history` if not already present
+- **On supervisor stop/complete:** Update latest run_history entry with `ended_at` and final `status`
+- **On feature.delete:** Archive run_history before deletion (optional)
+
+### 1.2 Operation Ledger Files
+
+**Location:** `.aop/runtime/operation-ledger/run:<run_id>.json`
+
+**Structure:**
+
+```json
+{
+  "run_id": "run:69c36d92-3255-435a-806a-6425461e5aa8",
+  "updated_at": "2026-03-05T06:07:17.975Z",
+  "operations": {
+    "<operation_id>": {
+      "operation_id": "feature_delete__my_feature__uuid",
+      "tool_name": "feature.delete",
+      "request_hash": "sha256...",
+      "response": {
+        "ok": true,
+        "data": { ... }
+      }
+    }
+  }
+}
+```
+
+**Key Fields:**
+
+- `operation_id`: Unique ID for idempotency tracking
+- `tool_name`: MCP tool invoked (e.g., `feature.init`, `plan.submit`, `gates.run`)
+- `request_hash`: SHA256 of request payload
+- `response`: Full tool response with `ok` and `data`/`error`
+
+### 1.3 Worker Events Files
+
+**Location:** `.aop/runtime/worker-events/run:<run_id>.jsonl`
+
+**Structure:** JSONL (one JSON object per line)
+
+```json
+{"ts":"2026-03-05T22:24:39.635Z","run_id":"run:8efe99b2...","feature_id":"my_feature","role":"planner","phase":"execution","event_type":"worker_started","output_types":[],"patch_count":0,"plan_submission_count":0,"request_count":0,"note_count":0,"valid":true,"error_code":null,"provider":"claude","model":"local-default","watchdog_reason":null,"signal":null}
+{"ts":"2026-03-05T22:25:17.441Z","run_id":"run:8efe99b2...","feature_id":"my_feature","role":"planner","phase":"execution","event_type":"worker_completed","output_types":["PLAN_SUBMISSION"],"patch_count":0,"plan_submission_count":1,"request_count":0,"note_count":0,"valid":true,"error_code":null,"provider":"claude","model":"local-default","watchdog_reason":null,"signal":null}
+```
+
+**Key Fields:**
+
+- `ts`: ISO 8601 timestamp
+- `feature_id`: Feature this event belongs to
+- `role`: Agent role (`planner`, `builder`, `qa`)
+- `event_type`: `worker_started`, `worker_completed`, `worker_failed`, `worker_timeout`
+- `valid`: Boolean indicating success/failure
+- `error_code`: Error code if `valid=false`
+- `watchdog_reason`: Timeout/failure reason
+
+---
+
+## 2. API Contract
+
+### 2.1 Operation Ledger Endpoint
+
+**Route:** `GET /api/runtime/operations`
+
+**Query Parameters:**
+
+- `feature_id` (required): Feature ID to filter operations
+- `run_id` (optional): Specific run ID (defaults to latest from `run_history`)
+- `project` (optional): Project name for multi-project setups
+- `limit` (optional, default 100): Max operations to return
+- `offset` (optional, default 0): Pagination offset
+
+**Run ID Resolution:**
+
+1. If `run_id` provided, use it directly
+2. Else read `.aop/features/<feature_id>/state.md`
+3. Extract `run_history[0].run_id` (latest run)
+4. If `run_history` is empty or missing, return `run_not_found`
+
+**Response Envelope:**
+
+```typescript
+{
+  ok: true,
+  data: {
+    run_id: string;
+    feature_id: string;
+    updated_at: string;
+    run_info: {
+      started_at: string;
+      ended_at: string | null;
+      status: 'active' | 'completed' | 'failed' | 'timeout';
+    };
+    operations: Array<{
+      operation_id: string;
+      tool_name: string;
+      request_hash: string;
+      timestamp: string; // extracted from operation_id or response
+      response: {
+        ok: boolean;
+        data?: unknown;
+        error?: {
+          code: string;
+          message: string;
+          details?: unknown;
+        };
+      };
+    }>;
+    total_count: number;
+    has_more: boolean;
+  },
+  metadata: {
+    source: 'runtime_ledger';
+  }
+}
+```
+
+**Error Codes:**
+
+- `feature_not_found`: Feature ID not found in index
+- `run_not_found`: No run_history found for feature
+- `ledger_not_found`: Operation ledger file missing for run_id
+- `ledger_parse_error`: JSON parse failure
+
+### 2.2 Worker Events Endpoint
+
+**Route:** `GET /api/runtime/events`
+
+**Query Parameters:**
+
+- `feature_id` (required): Feature ID to filter events
+- `run_id` (optional): Specific run ID (defaults to latest from `run_history`)
+- `project` (optional): Project name for multi-project setups
+- `limit` (optional, default 100): Max events to return
+- `offset` (optional, default 0): Pagination offset
+- `event_type` (optional): Filter by event type (comma-separated)
+- `role` (optional): Filter by role (comma-separated)
+
+**Run ID Resolution:**
+
+1. If `run_id` provided, use it directly
+2. Else read `.aop/features/<feature_id>/state.md`
+3. Extract `run_history[0].run_id` (latest run)
+4. If `run_history` is empty or missing, return `run_not_found`
+
+**Response Envelope:**
+
+```typescript
+{
+  ok: true,
+  data: {
+    run_id: string;
+    feature_id: string;
+    run_info: {
+      started_at: string;
+      ended_at: string | null;
+      status: 'active' | 'completed' | 'failed' | 'timeout';
+    };
+    events: Array<{
+      ts: string;
+      run_id: string;
+      feature_id: string;
+      role: string;
+      phase: string;
+      event_type: string;
+      output_types: string[];
+      patch_count: number;
+      plan_submission_count: number;
+      request_count: number;
+      note_count: number;
+      valid: boolean;
+      error_code: string | null;
+      provider: string;
+      model: string;
+      watchdog_reason: string | null;
+      signal: string | null;
+      exit_code?: number;
+      timeout_ms?: number;
+      elapsed_ms?: number;
+    }>;
+    total_count: number;
+    has_more: boolean;
+  },
+  metadata: {
+    source: 'runtime_events';
+  }
+}
+```
+
+**Error Codes:**
+
+- `feature_not_found`: Feature ID not found in index
+- `run_not_found`: No run_history found for feature
+- `events_not_found`: Worker events file missing for run_id
+- `events_parse_error`: JSONL parse failure
+
+### 2.3 Run History Endpoint (New)
+
+**Route:** `GET /api/runtime/runs`
+
+**Query Parameters:**
+
+- `feature_id` (required): Feature ID to get run history
+- `project` (optional): Project name for multi-project setups
+
+**Response Envelope:**
+
+```typescript
+{
+  ok: true,
+  data: {
+    feature_id: string;
+    runs: Array<{
+      run_id: string;
+      started_at: string;
+      ended_at: string | null;
+      status: 'active' | 'completed' | 'failed' | 'timeout';
+      has_operations: boolean;
+      has_events: boolean;
+    }>;
+  },
+  metadata: {
+    source: 'feature_state';
+  }
+}
+```
+
+**Purpose:** Allow UI to show run selector dropdown with all historical runs for a feature
+
+**Error Codes:**
+
+- `feature_not_found`: Feature ID not found in index
+- `state_not_found`: Feature state file missing
+- `state_parse_error`: YAML parse failure
+
+---
+
+## 3. UI Components
+
+### 3.1 Runtime Inspector Panel (New Component)
+
+**File:** `packages/web-dashboard/src/components/runtime-inspector.tsx`
+
+**Location:** Appears in Feature Detail Panel (right side) as a new tab
+
+**Tabs:**
+
+1. **Operations** - Operation Ledger view
+2. **Events** - Worker Events view
+
+**Run Selector:**
+
+- Dropdown at top of panel showing all runs from `run_history`
+- Format: `Run <short_id> - <started_at> - <status>`
+- Default: Latest run (index 0)
+- Fetches from `GET /api/runtime/runs?feature_id=<id>`
+
+**Default State:**
+
+- Operations tab selected
+- Latest run selected
+- Last 50 operations shown
+- Newest first (reverse chronological)
+- Raw JSON collapsed
+
+**Interaction:**
+
+- Select run from dropdown → refetch operations/events for that run
+- Click operation row to expand/collapse JSON
+- Copy button for operation_id and request_hash
+- Filter by tool_name (dropdown)
+- Filter by success/failure (toggle)
+- Pagination controls at bottom
+
+### 3.2 Operations List View
+
+**Requirements:**
+
+**Summary Row (Always Visible):**
+
+```
+[Icon] tool_name | timestamp | status_badge | expand_icon
+```
+
+**Status Badge:**
+
+- ✅ Success (green) - `response.ok === true`
+- ❌ Failed (red) - `response.ok === false`
+- ⚠️ Partial (yellow) - `response.ok === true` but `response.data` contains warnings
+
+**Expanded View:**
+
+```
+Operation ID: <operation_id> [Copy]
+Request Hash: <request_hash> [Copy]
+Tool: <tool_name>
+Timestamp: <formatted_timestamp>
+
+Response:
+<formatted_json_with_syntax_highlighting>
+```
+
+**Formatting:**
+
+- Use `JSON.stringify(response, null, 2)` for readability
+- Syntax highlighting for JSON (optional, use `<pre>` with CSS if no library)
+- Max height 400px with scroll for large responses
+
+### 3.3 Events Timeline View
+
+**Requirements:**
+
+**Timeline Row (Always Visible):**
+
+```
+[timestamp] [role_badge] [event_type_icon] event_type | duration | status
+```
+
+**Role Badge:**
+
+- 🎯 Planner (blue)
+- 🔨 Builder (orange)
+- ✅ QA (green)
+
+**Event Type Icons:**
+
+- ▶️ `worker_started`
+- ✅ `worker_completed`
+- ❌ `worker_failed`
+- ⏱️ `worker_timeout`
+
+**Duration Calculation:**
+
+- For `worker_completed`/`worker_failed`/`worker_timeout`: show `elapsed_ms` if present
+- For `worker_started`: calculate duration to next event for same role
+
+**Expanded View:**
+
+```
+Timestamp: <formatted_timestamp>
+Run ID: <run_id>
+Feature: <feature_id>
+Role: <role>
+Phase: <phase>
+Event Type: <event_type>
+Valid: <valid>
+Error Code: <error_code>
+Provider: <provider>
+Model: <model>
+Watchdog Reason: <watchdog_reason>
+Signal: <signal>
+Exit Code: <exit_code>
+Timeout: <timeout_ms>ms
+Elapsed: <elapsed_ms>ms
+
+Output Types: <output_types.join(', ')>
+Patch Count: <patch_count>
+Plan Submissions: <plan_submission_count>
+Requests: <request_count>
+Notes: <note_count>
+```
+
+### 3.4 Filter Controls
+
+**Operations Filters:**
+
+- Tool Name (dropdown, multi-select)
+- Status (Success/Failed/All)
+- Time Range (Last Hour/Last Day/All)
+
+**Events Filters:**
+
+- Role (Planner/Builder/QA/All)
+- Event Type (Started/Completed/Failed/Timeout/All)
+- Valid Only (toggle)
+- Time Range (Last Hour/Last Day/All)
+
+**Filter Persistence:**
+
+- Store in URL query params
+- Restore on page load
+- Clear filters button
+
+---
+
+## 4. Implementation Details
+
+### 4.0 Schema and State Management Changes
+
+**File:** `agentic/orchestrator/schemas/state.schema.json`
+
+**Add to properties:**
+
+```json
+{
+  "run_history": {
+    "type": "array",
+    "description": "Time-sorted list of run IDs associated with this feature (newest first). Updated by supervisor on run start/end.",
+    "default": [],
+    "items": {
+      "type": "object",
+      "required": ["run_id", "started_at"],
+      "additionalProperties": false,
+      "properties": {
+        "run_id": {
+          "type": "string",
+          "pattern": "^run:[a-f0-9-]+$",
+          "description": "Run ID from runtime_sessions"
+        },
+        "started_at": {
+          "type": "string",
+          "format": "date-time",
+          "description": "When this run started working on this feature"
+        },
+        "ended_at": {
+          "type": ["string", "null"],
+          "format": "date-time",
+          "description": "When this run stopped working on this feature (null if still active)"
+        },
+        "status": {
+          "type": "string",
+          "enum": ["active", "completed", "failed", "timeout"],
+          "description": "Final status of this run for this feature"
+        }
+      }
+    }
+  }
+}
+```
+
+**File:** `apps/control-plane/src/application/services/feature-state-service.ts`
+
+**Changes:**
+
+- Add `appendRunHistory(featureId, runId, startedAt)` method
+- Add `updateRunHistory(featureId, runId, endedAt, status)` method
+- Ensure run_history is sorted newest-first on write
+
+**File:** `apps/control-plane/src/supervisor/runtime.ts`
+
+**Changes:**
+
+- On feature start: Call `featureStateService.appendRunHistory(featureId, this.runId, new Date().toISOString())`
+- On feature complete/fail: Call `featureStateService.updateRunHistory(featureId, this.runId, new Date().toISOString(), status)`
+- On recovery: Check if current run_id exists in run_history, append if missing
+
+**Migration Strategy:**
+
+- Existing features without run_history: Initialize as empty array `[]`
+- No data loss: run_history is additive only
+- Backward compatible: Old state files without run_history still validate (default: `[]`)
+
+### 4.1 Backend Implementation
+
+**File:** `packages/web-dashboard/src/app/api/runtime/runs/route.ts` (New)
+
+**Logic:**
+
+1. Resolve `feature_id` and `project` from query params
+2. Read `.aop/features/<feature_id>/state.md`
+3. Parse YAML frontmatter to extract `run_history`
+4. For each run, check if ledger/events files exist
+5. Return formatted response with run list
+
+**File:** `packages/web-dashboard/src/app/api/runtime/operations/route.ts`
+
+**Logic:**
+
+1. Resolve `feature_id` and `project` from query params
+2. If `run_id` provided, use it; else read state.md and extract `run_history[0].run_id`
+3. If no run_history, return `run_not_found`
+4. Read `.aop/runtime/operation-ledger/run:<run_id>.json`
+5. Parse JSON and extract operations
+6. Filter operations by `feature_id` (check operation_id contains feature_id)
+7. Sort by timestamp (extract from operation_id or use file `updated_at`)
+8. Apply pagination (limit/offset)
+9. Return formatted response with run_info from run_history
+
+**File:** `packages/web-dashboard/src/app/api/runtime/events/route.ts`
+
+**Logic:**
+
+1. Resolve `feature_id` and `project` from query params
+2. If `run_id` provided, use it; else read state.md and extract `run_history[0].run_id`
+3. If no run_history, return `run_not_found`
+4. Read `.aop/runtime/worker-events/run:<run_id>.jsonl`
+5. Parse JSONL (split by newline, parse each line as JSON)
+6. Filter events by `feature_id`
+7. Apply filters (event_type, role, valid)
+8. Sort by `ts` (descending for newest first)
+9. Apply pagination (limit/offset)
+10. Return formatted response with run_info from run_history
+
+**Error Handling:**
+
+- Catch file read errors → return `ledger_not_found` or `events_not_found`
+- Catch JSON parse errors → return `ledger_parse_error` or `events_parse_error`
+- Handle missing run_history → return `run_not_found`
+- Handle large files (> 10MB) → return `file_too_large` with suggestion to use CLI
+
+### 4.2 Frontend Implementation
+
+**File:** `packages/web-dashboard/src/components/runtime-inspector.tsx`
+
+**State Management:**
+
+```typescript
+interface RuntimeInspectorState {
+  activeTab: 'operations' | 'events';
+  selectedRunId: string | null;
+  runs: RunHistoryEntry[];
+  operations: Operation[];
+  events: WorkerEvent[];
+  loading: boolean;
+  error: string | null;
+  filters: {
+    toolName?: string[];
+    status?: 'success' | 'failed' | 'all';
+    eventType?: string[];
+    role?: string[];
+    validOnly?: boolean;
+    timeRange?: 'hour' | 'day' | 'all';
+  };
+  pagination: {
+    limit: number;
+    offset: number;
+    totalCount: number;
+    hasMore: boolean;
+  };
+  expandedIds: Set<string>;
+}
+
+interface RunHistoryEntry {
+  run_id: string;
+  started_at: string;
+  ended_at: string | null;
+  status: 'active' | 'completed' | 'failed' | 'timeout';
+  has_operations: boolean;
+  has_events: boolean;
+}
+```
+
+**Hooks:**
+
+- `useRunHistory(featureId)` - Fetch run history list
+- `useRuntimeOperations(featureId, runId, filters, pagination)` - Fetch operations
+- `useRuntimeEvents(featureId, runId, filters, pagination)` - Fetch events
+- `useExpandedState()` - Manage expanded/collapsed rows
+
+**Component Lifecycle:**
+
+1. Mount → fetch run history
+2. Run history loaded → select latest run (index 0)
+3. Run selected → fetch operations/events for that run
+4. Run selector change → refetch operations/events
+
+**Performance:**
+
+- Virtualize lists if > 100 items (use `react-window` or similar)
+- Debounce filter changes (300ms)
+- Cancel in-flight requests on run/filter change
+- Cache responses per (feature_id, run_id) pair (60s TTL)
+
+### 4.3 Integration with Detail Panel
+
+**File:** `packages/web-dashboard/src/components/detail-panel.tsx`
+
+**Changes:**
+
+1. Add new tab: "Runtime" (after "Evidence" tab)
+2. Render `<RuntimeInspector featureId={selectedFeature.feature_id} />` when tab is active
+3. Badge on tab shows total operation count (if available)
+
+**Tab Order:**
+
+1. Overview
+2. Plan
+3. Gates
+4. Evidence
+5. **Runtime** (new)
+6. Logs
+
+---
+
+## 5. Testing Strategy
+
+### 5.1 Unit Tests
+
+**Schema Tests:**
+
+- `schemas/state.schema.spec.ts`
+  - Validates run_history field structure
+  - Rejects invalid run_history entries
+  - Allows empty run_history array
+  - Validates date-time formats
+
+**Backend Tests:**
+
+- `api/runtime/runs/route.spec.ts`
+  - Returns run history for valid feature_id
+  - Returns 404 for missing feature
+  - Returns 404 for missing state file
+  - Handles YAML parse errors
+  - Checks file existence for operations/events
+
+- `api/runtime/operations/route.spec.ts`
+  - Returns operations for valid feature_id
+  - Uses latest run_id when not specified
+  - Uses specified run_id when provided
+  - Returns 404 for missing feature
+  - Returns 404 for missing run_history
+  - Returns 404 for missing ledger file
+  - Handles JSON parse errors
+  - Applies pagination correctly
+  - Filters by tool_name
+  - Sorts by timestamp descending
+
+- `api/runtime/events/route.spec.ts`
+  - Returns events for valid feature_id
+  - Uses latest run_id when not specified
+  - Uses specified run_id when provided
+  - Returns 404 for missing feature
+  - Returns 404 for missing run_history
+  - Returns 404 for missing events file
+  - Handles JSONL parse errors
+  - Applies pagination correctly
+  - Filters by event_type and role
+  - Sorts by timestamp descending
+
+**Frontend Tests:**
+
+- `components/runtime-inspector.spec.tsx`
+  - Renders operations tab by default
+  - Fetches run history on mount
+  - Selects latest run by default
+  - Switches between tabs
+  - Changes selected run
+  - Refetches data on run change
+  - Expands/collapses operation rows
+  - Applies filters
+  - Handles pagination
+  - Shows loading state
+  - Shows error state
+  - Copies operation_id to clipboard
+
+### 5.2 Integration Tests
+
+**End-to-End:**
+
+1. Create feature with multiple runs
+2. Open feature detail panel
+3. Click "Runtime" tab
+4. Verify run selector appears with all runs
+5. Verify latest run is selected
+6. Verify operations list appears
+7. Select older run from dropdown
+8. Verify operations list updates
+9. Expand operation row
+10. Verify JSON response is visible
+11. Switch to Events tab
+12. Verify events timeline appears
+13. Apply filter (e.g., "Failed only")
+14. Verify filtered results
+
+### 5.3 Performance Tests
+
+**Benchmarks:**
+
+- Load run history (10 runs) in < 50ms
+- Load 100 operations in < 200ms
+- Load 100 events in < 200ms
+- Render 100 operations in < 100ms
+- Filter 100 operations in < 50ms
+- Pagination in < 50ms
+- Run selector change in < 100ms
+
+**Stress Tests:**
+
+- Handle 50 runs in history gracefully
+- Handle 1000 operations gracefully (virtualization)
+- Handle 1000 events gracefully (virtualization)
+- Handle 10MB ledger file (return error with suggestion)
+
+---
+
+## 6. Acceptance Criteria
+
+### 6.1 Must Have
+
+- [ ] Schema change: Add `run_history` field to `state.schema.json`
+- [ ] Supervisor: Update run_history on feature start/stop
+- [ ] FeatureStateService: Support run_history read/write
+- [ ] `GET /api/runtime/runs` endpoint implemented
+- [ ] `GET /api/runtime/operations` endpoint implemented
+- [ ] `GET /api/runtime/events` endpoint implemented
+- [ ] Run ID resolution from run_history (latest by default)
+- [ ] RuntimeInspector component renders in detail panel
+- [ ] Run selector dropdown with all historical runs
+- [ ] Operations tab shows last 50 operations by default
+- [ ] Events tab shows last 50 events by default
+- [ ] Operations sorted by timestamp (newest first)
+- [ ] Events sorted by timestamp (newest first)
+- [ ] Expand/collapse operation rows
+- [ ] Expand/collapse event rows
+- [ ] Copy operation_id and request_hash
+- [ ] Filter operations by tool_name
+- [ ] Filter operations by success/failure
+- [ ] Filter events by role
+- [ ] Filter events by event_type
+- [ ] Pagination controls (Next/Previous)
+- [ ] Loading state
+- [ ] Error state with retry button
+- [ ] Empty state ("No operations recorded")
+- [ ] All tests pass (>= 90% coverage)
+- [ ] TypeScript strict mode passes
+- [ ] ESLint zero warnings
+
+### 6.2 Should Have
+
+- [ ] Syntax highlighting for JSON responses
+- [ ] Duration calculation for events
+- [ ] Time range filters (Last Hour/Last Day/All)
+- [ ] Export operations to JSON file
+- [ ] Export events to JSON file
+- [ ] Search within operation responses
+- [ ] Keyboard navigation (arrow keys to expand/collapse)
+
+### 6.3 Nice to Have
+
+- [ ] Real-time updates via SSE (append new operations/events)
+- [ ] Diff view for plan.update operations
+- [ ] Link from operation to related feature state change
+- [ ] Flame graph for operation timing
+- [ ] Event correlation (link worker_started to worker_completed)
+
+---
+
+## 7. Non-Goals
+
+- **Mutation of runtime files:** This feature is read-only inspection
+- **Global runtime view:** Always scoped to single feature
+- **Historical run comparison:** Only shows latest run (or specified run_id)
+- **Real-time streaming:** Initial version uses polling/manual refresh
+- **Advanced analytics:** No aggregation, charts, or trend analysis (use Analytics page)
+
+---
+
+## 8. Rollout Plan
+
+### 8.1 Phase 0: Schema and State Management (Week 1)
+
+- Add `run_history` field to `state.schema.json`
+- Update FeatureStateService to read/write run_history
+- Update SupervisorRuntime to append run_id on feature start
+- Update SupervisorRuntime to update run status on feature end
+- Add migration for existing features (backfill empty run_history)
+- Add unit tests for schema validation
+- Add unit tests for state service
+
+### 8.2 Phase 1: Backend API (Week 2)
+
+- Implement `/api/runtime/runs` endpoint
+- Implement `/api/runtime/operations` endpoint with run_history resolution
+- Implement `/api/runtime/events` endpoint with run_history resolution
+- Add unit tests
+- Add integration tests
+- Document API in README
+
+### 8.3 Phase 2: Frontend UI (Week 3)
+
+- Implement RuntimeInspector component
+- Add run selector dropdown
+- Add Operations tab
+- Add Events tab
+- Add filter controls
+- Add pagination
+- Integrate with detail panel
+
+### 8.4 Phase 3: Polish (Week 4)
+
+- Add syntax highlighting
+- Add duration calculations
+- Add export functionality
+- Add keyboard navigation
+- Performance optimization
+- Accessibility audit
+
+### 8.5 Phase 4: Documentation (Week 4)
+
+- Update state.schema.json documentation
+- Update dashboard README
+- Add screenshots to docs
+- Create user guide
+- Update CLAUDE.md and AGENTS.md
+
+---
+
+## 9. Success Metrics
+
+### 9.1 Functional Metrics
+
+- API response time < 200ms (p95)
+- UI render time < 100ms (p95)
+- Zero crashes on large files (< 10MB)
+- 100% test coverage for new code
+
+### 9.2 User Metrics
+
+- Time to find failure cause reduced by 50%
+- Number of "What happened?" questions reduced
+- Operator satisfaction score > 4/5
+
+---
+
+## 10. References
+
+### 10.1 Related Documents
+
+- `spec-files/completed/agentic_orchestrator_dashboard_advanced_ux_spec.md` - Dashboard UX standards
+- `README.md` - CLI and tool contracts
+- `CLAUDE.md` - Architecture patterns
+
+### 10.2 Related Code
+
+- `packages/web-dashboard/src/components/detail-panel.tsx` - Feature detail panel
+- `packages/web-dashboard/src/lib/aop-client.ts` - API client utilities
+- `apps/control-plane/src/mcp/tool-runtime.ts` - Operation ledger writer
+- `apps/control-plane/src/supervisor/runtime.ts` - Worker events writer
+
+---
+
+**End of Specification**