pi-crew 0.1.49 → 0.2.0

package/docs/research-phase9-observability-reliability-plan.md

@@ -1,1190 +0,0 @@
-# Phase 9 — Observability & Reliability (Theme B + C combined)
-
-> Path X: Phase 8 (Operator Experience) → **Phase 9 (Observability + Reliability)**. Mục tiêu: build telemetry backbone (Counter/Gauge/Histogram + correlation ID + sink/export) đồng thời harden run reliability (heartbeat gradient + retry + crash recovery + deadletter). Combined vì 5 synergy critical (xem mục 1.A).
-
-> **Prerequisite:** Phase 8 đã DONE (verified 351 unit + 44 integration pass, version 0.1.34) — `NotificationRouter`, `ConfirmOverlay`, `MailboxDetailOverlay/Compose/Preview/AgentPicker`, `heartbeat-aggregator.ts`, `health-pane.ts`, `diagnostic-export.ts` (with `redactSecrets` regex `/(token|key|password|secret|credential|auth)/i`), `notification-sink.ts`, `keybinding-map.ts`, `run-action-dispatcher.ts` — Phase 9 reuse.
-
-> **Critical preflight finding (Phase 9.0.E):** `ExtensionAPI.events` interface is `EventBus` from `pi-coding-agent/dist/core/event-bus.d.ts`:
-> ```ts
-> interface EventBus { emit(channel, data): void; on(channel, handler): () => void; }  // on() returns unsubscribe function — NO off() method
-> ```
-> → All "dispose" patterns must capture `unsubscribe` from `on()` return value, NOT call `events.off()`.
-
-## 0. Implementation Status
-
-### Foundation (Wave 1)
-- [x] 9.0.A Metric primitives — Counter / Gauge / Histogram base classes (`src/observability/metrics-primitives.ts`)
-- [x] 9.0.B MetricRegistry **per-session instance** + naming convention (`src/observability/metric-registry.ts`)
-- [x] 9.0.C Correlation context — traceId/spanId propagation primitive (`src/observability/correlation.ts`)
-- [x] 9.0.D Heartbeat gradient classifier extension (warn/stale/dead thresholds with metrics emission, reuse `WorkerHeartbeatState` interface + `isWorkerHeartbeatStale` helper)
-- [x] 9.0.E **Preflight verify** ExtensionAPI surface (`events.on` returns unsubscribe fn, `events.off` does NOT exist) + cross-check `WorkerHeartbeatState` field name
-
-### Reliability core (Wave 2)
-- [x] 9.1.A Background heartbeat watcher (detect stuck workers, emit `crew.heartbeat.staleness_ms` Gauge)
-- [x] 9.1.B Retry executor + backoff/jitter policy (`src/runtime/retry-executor.ts`)
-- [x] 9.1.C Crash recovery resume từ event-log checkpoint
-- [x] 9.1.D Deadletter queue writer + threshold alerts via NotificationRouter
-
-### Telemetry pipeline (Wave 3)
-- [x] 9.2.A Event-to-metric subscriber (subscribe `crew.*` events → registry counters)
-- [x] 9.2.B Metric retention policy (sliding window aggregation 1h/1d configurable)
-- [x] 9.2.C Histogram quantile calculator (p50/p95/p99 streaming) — t-digest or fixed buckets
-- [x] 9.2.D Metric file sink JSONL với daily rotation (gated bởi `telemetry.enabled`)
-
-### Export adapters (Wave 3 parallel)
-- [x] 9.3.A Prometheus exposition format adapter (HTTP endpoint optional)
-- [x] 9.3.B OTLP HTTP exporter (optional, opt-in)
-- [x] 9.3.C Adapter abstraction (plugin pattern, extensible)
-
-### UI & commands (Wave 4)
-- [x] 9.4.A `team metrics` command — snapshot JSON, filter by name/runId
-- [x] 9.4.B Metrics pane (pane index `6`) trong dashboard
-- [x] 9.4.C Diagnostic export (Phase 8) include metrics snapshot
-
-### Wiring & validation (Wave 5)
-- [x] 9.5.A Wire register.ts — instantiate MetricRegistry, EventToMetric subscriber, RetryExecutor, BackgroundWatcher
-- [x] 9.5.B Tests: unit + integration + perf
-- [x] 9.5.C Migration guide: existing runs continue to work; opt-in for retry/recovery via config flag
-
-## 1. Roadmap-Level Decisions
-
-### 1.A Synergy Theme B + C — 5 critical integrations
-
-| # | Touchpoint | Theme B contributes | Theme C contributes | Combined value |
-|---|---|---|---|---|
-| **S1** | Heartbeat staleness | Gauge primitive `crew.heartbeat.staleness_ms{runId,taskId}` | Gradient classifier (healthy/warn/stale/dead) | Auto-emit metric per task → time-series → detect regression |
-| **S2** | Retry attempts | Histogram primitive `crew.task.retry_count{team}` | Retry executor + jitter backoff | Distribution analytics (p95 retries per team) |
-| **S3** | Recovery trace | `traceId`/`spanId` correlation propagation | Recovery state machine (resume từ checkpoint) | Cross-component debug — subagent crash → recovery → resume fully traceable |
-| **S4** | Deadletter alert | Counter `crew.task.deadletter_total{reason}` + threshold | Deadletter writer | Auto-alert via NotificationRouter khi rate > threshold |
-| **S5** | Performance regression | Histogram quantile p95 over time | Stale duration tracking | Detect "Phase X deploy → p95 staleness +50%" tự động |
-
-### 1.B Decisions
-
-| # | Decision | Chosen | Rationale |
-|---|---|---|---|
-| D1 | Metric primitives: implement custom hay reuse library? | **Implement custom (minimal)** — Counter, Gauge, Histogram chỉ ~200 LOC | Tránh dependency mới (đồng nhất Phase 7/8 zero-dep approach); OTLP serializer cũng < 200 LOC |
-| D2 | Histogram bucket strategy? | **Fixed exponential buckets** `[1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]` ms | Simple, predictable; no t-digest complexity; 95% use case là latency ms; user override qua config nếu cần |
-| D3 | Correlation ID format? | **`{runId}:{taskId}:{spanCounter}`** (P1 default) | Human-readable, không cần UUID library, deterministic cho test, scope rõ ràng |
-| D4 | Correlation ID propagation method? | **Async context (`AsyncLocalStorage`)** trong Node.js runtime | Standard Node API; không phải pass thủ công qua mọi function; minimal overhead |
-| D5 | Retry executor: opt-in hay default-on? | **Opt-in** qua `reliability.autoRetry: false` mặc định | Risk High (touches state machine); user explicit consent; preserve current behavior bằng default |
-| D6 | Retry policy default? | **maxAttempts=3, backoffMs=1000, jitterRatio=0.3, exponentialFactor=2** (P2) | Sensible defaults; per-task override; matches industry common pattern |
-| D7 | Crash recovery: auto-resume vs prompt? | **Prompt via NotificationRouter** (P3) — Phase 8 ConfirmOverlay reused | User confirmation cho destructive resume action; tránh false-positive replay |
-| D8 | Metric retention window default? | **1 hour streaming, 24 hour summary** (P4); persist daily JSONL | Cover 95% debugging; balance memory vs disk |
-| D9 | Background watcher polling interval? | **5 seconds** default, configurable 1-60s (P8) | Responsive without burn CPU; setInterval not setTimeout chain |
-| D10 | OTLP export priority? | **Implement nhưng disable mặc định** (P6) | Foundation cho team có observability stack; off by default tránh confused user |
-| D11 | Deadletter alert threshold? | **>3 deadletter messages trong 1 hour** (P7) | Conservative; tránh false positive; configurable |
-| D12 | Event-to-metric mapping cấu hình hay hardcode? | **Hardcode core** + extensible plugin | Core ~15 events đã định, hardcode đảm bảo consistent; plugin cho user custom |
-| D13 | Naming convention metrics? | **`crew.{domain}.{measure}_{unit}`** — `crew.run.duration_ms`, `crew.task.retry_count`, `crew.heartbeat.staleness_ms` | Prometheus-compatible; domain rõ ràng; unit suffix tránh ambiguity |
-| D14 | Metric sink file location? | **`<crewRoot>/state/metrics/{YYYY-MM-DD}.jsonl`** | Đồng nhất với Phase 8 notification sink pattern; daily rotation; configurable retention |
-| D15 | Recovery checkpoint format? | **Event-log cursor** (existing `events.jsonl.seq` + `sequencePath()`/`scanSequence()` helpers) | Reuse hạ tầng đã có Phase 6; không thêm checkpoint format mới |
-| D16 | Histogram quantile algorithm? | **Fixed buckets + linear interpolation** (P5) | Đơn giản; sufficient cho p50/p95/p99 với fixed buckets; t-digest defer Phase 10 nếu cần |
-| **D17** | **MetricRegistry lifecycle** | **Per-session instance** (consistent với Phase 8 `notificationRouter`/`heartbeatAggregator`) — instantiate trong `session_start`, `dispose()` trong `session_shutdown` | Cumulative metrics across sessions không cần thiết Phase 9 (defer Phase 10 nếu user yêu cầu); test isolation tự nhiên; no global state leak; dispose semantics rõ ràng |
-| **D18** | **Event subscription cleanup** | **Capture unsubscribe fn từ `events.on()` return value**; KHÔNG call `events.off()` (không tồn tại trên `EventBus` interface) | API surface preflight verified (9.0.E); pattern matches existing usages trong codebase (`src/ui/render-scheduler.ts`) |
-| **D19** | **Retry state machine semantics** | **Task `failed` chỉ transition khi maxAttempts exhausted**; thêm field `task.attempts: Array<{startedAt,endedAt,error?}>` cho traceability; artifact final chỉ trên terminal attempt | Tránh terminal-state monotonicity violation (re-run task đang `failed` về `running`); audit trail đầy đủ cho debug |
-| **D20** | **Crash recovery trigger combinator** | Recovery only triggers if `(status==="running") AND (no async.pid OR async.pid is dead via existing liveness check) AND (heartbeat dead via isWorkerHeartbeatStale > deadMs OR no heartbeat)` | Tránh false-positive marking healthy async run là interrupted; reuse Phase 6/7 async.pid liveness check trong `session-summary.ts` |
-| **D21** | **Diagnostic schema versioning** | `DiagnosticReport.schemaVersion: 2` khi thêm `metricsSnapshot?: MetricSnapshot[]` field; apply `redactSecrets()` recursive trên `metricsSnapshot` (label values có thể chứa secret patterns) | Backward-compat consumer reading old format (schemaVersion missing → treat as v1); secret leak prevention |
-| **D22** | **Deadletter trigger separation** | 3 paths: (a) `executeWithRetry` exhaust → write entry; (b) heartbeat watcher dead 3 ticks consecutive → write entry; (c) Counter rate > 3/hour → NotificationRouter alert | Trigger entry vs threshold alert là 2 logic riêng; tránh conflate trong implementation |
-
-## 2. Phase Breakdown
-
-### Phase 9.0 — Foundation (3.5 dev-days)
-
-#### 9.0.A Metric primitives (1 dev-day)
-
-**File mới:** `src/observability/metrics-primitives.ts`
-
-```ts
-export interface MetricLabels {
-	[key: string]: string | number;
-}
-
-export abstract class Metric {
-	constructor(public readonly name: string, public readonly description: string) {}
-	abstract snapshot(): MetricSnapshot;
-}
-
-export class Counter extends Metric {
-	private values = new Map<string, number>();  // labelKey → count
-	inc(labels: MetricLabels = {}, delta = 1): void { /* ... */ }
-	snapshot(): MetricSnapshot { return { type: "counter", name: this.name, values: [...this.values.entries()] }; }
-}
-
-export class Gauge extends Metric {
-	private values = new Map<string, number>();
-	set(labels: MetricLabels, value: number): void { /* ... */ }
-	add(labels: MetricLabels, delta: number): void { /* ... */ }
-	snapshot(): MetricSnapshot { /* ... */ }
-}
-
-export class Histogram extends Metric {
-	private buckets: number[];  // upper bounds, e.g. [1, 5, 10, 25, ...]
-	private observations = new Map<string, { counts: number[]; sum: number; count: number }>();
-	constructor(name: string, description: string, buckets?: number[]) {
-		super(name, description);
-		this.buckets = buckets ?? [1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000];
-	}
-	observe(labels: MetricLabels, value: number): void { /* ... */ }
-	quantile(labels: MetricLabels, q: number): number { /* linear interpolation */ }
-	snapshot(): MetricSnapshot { /* ... */ }
-}
-
-export interface MetricSnapshot {
-	type: "counter" | "gauge" | "histogram";
-	name: string;
-	values: unknown;
-}
-```
-
-**Tests:** `test/unit/metrics-primitives.test.ts` — 12 cases (counter inc/labels, gauge set/add/labels, histogram observe/quantile p50/p95/p99/edge empty/edge single value).
-
-#### 9.0.B MetricRegistry (0.75 dev-day) — **Per-session instance (D17)**
-
-**File mới:** `src/observability/metric-registry.ts`
-
-```ts
-export class MetricRegistry {
-	private metrics = new Map<string, Metric>();
-	registerCounter(name: string, description: string): Counter { /* ... */ }
-	registerGauge(name: string, description: string): Gauge { /* ... */ }
-	registerHistogram(name: string, description: string, buckets?: number[]): Histogram { /* ... */ }
-	get(name: string): Metric | undefined { return this.metrics.get(name); }
-	snapshot(): MetricSnapshot[] { return [...this.metrics.values()].map((m) => m.snapshot()); }
-	dispose(): void { this.metrics.clear(); }
-}
-
-// Per-session factory — caller (register.ts) instantiates trong session_start, dispose trong session_shutdown.
-// KHÔNG dùng singleton pattern (xem D17): tránh state leak cross-session, đảm bảo test isolation.
-export function createMetricRegistry(): MetricRegistry { return new MetricRegistry(); }
-```
-
-**Naming convention enforce (D13):** `name` phải match regex `^crew\.[a-z]+\.[a-z][a-z_]*$` (đơn giản hơn regex cũ `^crew\.[a-z_]+\.[a-z_]+(_[a-z]+)?$` vốn redundant). Unit suffix là phần của measure name (e.g., `duration_ms`, `staleness_ms`). Throw nếu không match.
-
-**Tests:** `test/unit/metric-registry.test.ts` — 6 cases (register, duplicate throws, snapshot all, naming validation, dispose clears state, get returns undefined sau dispose).
-
-#### 9.0.C Correlation context (1 dev-day)
-
-**File mới:** `src/observability/correlation.ts`
-
-```ts
-import { AsyncLocalStorage } from "node:async_hooks";
-
-export interface CorrelationContext {
-	traceId: string;       // {runId}:{taskId}:{spanCounter}
-	parentSpanId?: string;
-	spanId: string;
-}
-
-const storage = new AsyncLocalStorage<CorrelationContext>();
-let spanCounter = 0;
-
-export function withCorrelation<T>(ctx: CorrelationContext, fn: () => T): T {
-	return storage.run(ctx, fn);
-}
-
-export function getCurrentContext(): CorrelationContext | undefined {
-	return storage.getStore();
-}
-
-export function newSpanId(runId: string, taskId?: string): string {
-	spanCounter++;
-	return `${runId}:${taskId ?? "main"}:${spanCounter}`;
-}
-
-// Wrap event emission to inject correlation
-export function correlatedEvent<T extends { runId?: string; data?: Record<string, unknown> }>(event: T): T {
-	const ctx = getCurrentContext();
-	if (!ctx) return event;
-	return { ...event, data: { ...event.data, traceId: ctx.traceId, spanId: ctx.spanId, parentSpanId: ctx.parentSpanId } };
-}
-```
-
-**Wire vào `register.ts`** trong `pi.events.emit` wrapper — tất cả `crew.*` events tự inject correlation nếu context active. Foreground/async run wrap toàn bộ executeTeamRun trong `withCorrelation({traceId, spanId: newSpanId(runId)})`.
-
-**Tests:** `test/unit/correlation.test.ts` — 5 cases (basic propagation, nested span, missing context graceful, async boundary preserve, parallel runs isolated).
-
-#### 9.0.D Heartbeat gradient classifier (0.75 dev-day)
-
-**File mới:** `src/runtime/heartbeat-gradient.ts`
-
-```ts
-import type { WorkerHeartbeatState } from "./worker-heartbeat.ts";  // Phase 6/7 file — actual interface name (NOT "WorkerHeartbeat")
-
-export type HeartbeatLevel = "healthy" | "warn" | "stale" | "dead";
-
-export interface GradientThresholds {
-	warnMs: number;     // default 30_000 (30s)
-	staleMs: number;    // default 60_000 (1min)
-	deadMs: number;     // default 300_000 (5min)
-}
-
-export const DEFAULT_GRADIENT_THRESHOLDS: GradientThresholds = { warnMs: 30_000, staleMs: 60_000, deadMs: 300_000 };
-
-export function classifyHeartbeat(heartbeat: WorkerHeartbeatState | undefined, thresholds: GradientThresholds = DEFAULT_GRADIENT_THRESHOLDS, now = Date.now()): HeartbeatLevel {
-	if (!heartbeat) return "dead";
-	if (heartbeat.alive === false) return "dead";
-	const lastSeen = Date.parse(heartbeat.lastSeenAt);
-	if (!Number.isFinite(lastSeen)) return "dead";
-	const elapsed = now - lastSeen;
-	if (elapsed >= thresholds.deadMs) return "dead";
-	if (elapsed >= thresholds.staleMs) return "stale";
-	if (elapsed >= thresholds.warnMs) return "warn";
-	return "healthy";
-}
-```
-
-**Update `src/ui/heartbeat-aggregator.ts`** (Phase 8 file, 1612 bytes — verified existence) — backward-compat strategy:
-- Giữ nguyên existing API surface `summarizeHeartbeats(snapshot, opts)` returning `HeartbeatSummary` (Phase 8 caller `health-pane.ts` không break).
-- Internal classify SWITCH sang `classifyHeartbeat`; map 4-level (healthy/warn/stale/dead) → existing 3-bucket count (`healthy`/`stale`/`dead` — `warn` count merge vào `healthy` để giữ Phase 8 semantics).
-- Optional new field `summary.gradient: { healthy, warn, stale, dead }` cho consumers Phase 9 (metrics-pane).
-- Emit metrics khi `registry` param truyền vào (optional, không break Phase 8 caller):
-  - `metrics.gauge("crew.heartbeat.staleness_ms").set({runId, taskId}, elapsed)`
-  - `metrics.counter("crew.heartbeat.level_total").inc({runId, level})`
-
-**Tests:** `test/unit/heartbeat-gradient.test.ts` — 8 cases (healthy/warn/stale/dead/missing/explicit-dead/edge-now/custom-thresholds + invalid date string returns dead).
-
-#### 9.0.E Preflight ExtensionAPI surface verify (0.5 dev-day) — **NEW**
-
-**Mục tiêu:** Trước khi Wave 2 wire `events?.on?.()` callbacks, confirm bằng test tự động:
-
-**File mới:** `test/unit/extension-api-surface.test.ts` — verify hợp đồng:
-1. `pi.events.on(channel, handler)` returns function (unsubscribe).
-2. Calling unsubscribe stops handler invocation on subsequent emit.
-3. Multiple `on()` calls cho cùng channel đều được gọi.
-4. Confirm `events.off` không tồn tại (typeof check) — fail-fast nếu Pi upstream thay đổi API.
-5. Verify `WorkerHeartbeatState` interface fields exist (`workerId`, `lastSeenAt`, `alive?`) — guard against rename.
-
-**Output:** Block Wave 2 nếu test fail. Document trong PR description.
-
-**Tests:** chính là content của file 9.0.E (5 cases).
-
----
-
-### Phase 9.1 — Reliability Core (5 dev-days)
-
-#### 9.1.A Background heartbeat watcher (1.5 dev-days)
-
-**File mới:** `src/runtime/heartbeat-watcher.ts`
-
-**Logic:** Setup `setInterval(5000ms)` (D9) trong session_start; mỗi tick, đọc tất cả active runs từ `manifestCache.list(50)`, load tasks via `loadRunManifestById(cwd, runId).tasks`, classify mỗi task heartbeat:
-- `dead` lần đầu detect → emit `crew.task.heartbeat_dead` event + Counter `crew.heartbeat.dead_total{runId}` inc + NotificationRouter alert (severity warning, dedup id `dead_${runId}_${taskId}`).
-- `dead` consecutive 3 ticks → trigger deadletter writer (xem 9.1.D path b — D22).
-
-**Skeleton:**
-
-```ts
-import { loadRunManifestById } from "../state/state-store.ts";
-import type { WorkerHeartbeatState } from "./worker-heartbeat.ts";  // actual interface name
-import { classifyHeartbeat, DEFAULT_GRADIENT_THRESHOLDS, type HeartbeatLevel } from "./heartbeat-gradient.ts";
-
-export class HeartbeatWatcher {
-	private timer?: ReturnType<typeof setInterval>;
-	private lastLevel = new Map<string, HeartbeatLevel>();    // `${runId}:${taskId}` → previous level
-	private consecutiveDead = new Map<string, number>();      // `${runId}:${taskId}` → consecutive dead tick count
-	constructor(
-		private opts: {
-			cwd: string;
-			pollIntervalMs?: number;
-			thresholds?: GradientThresholds;
-			manifestCache: ManifestCache;
-			registry: MetricRegistry;
-			router: NotificationRouter;
-			deadletterTickThreshold?: number;  // default 3 (D22 path b)
-			onDead?: (runId: string, taskId: string, elapsed: number) => void;
-			onDeadletterTrigger?: (runId: string, taskId: string) => void;
-		}
-	) {}
-	start(): void {
-		this.timer = setInterval(() => this.tick(), this.opts.pollIntervalMs ?? 5000);
-	}
-	private tick(): void {
-		const thresholds = this.opts.thresholds ?? DEFAULT_GRADIENT_THRESHOLDS;
-		const tickThreshold = this.opts.deadletterTickThreshold ?? 3;
-		for (const run of this.opts.manifestCache.list(50)) {
-			if (run.status !== "running") continue;
-			const loaded = loadRunManifestById(this.opts.cwd, run.runId);
-			if (!loaded) continue;
-			for (const task of loaded.tasks) {
-				if (task.status !== "running" && task.status !== "queued") continue;
-				const key = `${run.runId}:${task.id}`;
-				const level = classifyHeartbeat(task.heartbeat, thresholds);
-				const prev = this.lastLevel.get(key);
-				this.lastLevel.set(key, level);
-				if (level === "dead" && prev !== "dead") {
-					this.opts.router.enqueue({ id: `dead_${run.runId}_${task.id}`, severity: "warning", source: "heartbeat-watcher", runId: run.runId, title: `Task ${task.id} heartbeat dead`, body: "Background watcher detected stuck worker." });
-					this.opts.registry.get("crew.heartbeat.dead_total")?.inc({ runId: run.runId });
-					this.opts.onDead?.(run.runId, task.id, 0);
-				}
-				if (level === "dead") {
-					const count = (this.consecutiveDead.get(key) ?? 0) + 1;
-					this.consecutiveDead.set(key, count);
-					if (count === tickThreshold) this.opts.onDeadletterTrigger?.(run.runId, task.id);
-				} else this.consecutiveDead.delete(key);
-			}
-		}
-	}
-	dispose(): void {
-		if (this.timer) clearInterval(this.timer);
-		this.timer = undefined;
-		this.lastLevel.clear();
-		this.consecutiveDead.clear();
-	}
-}
-```
-
-**Tests:** `test/unit/heartbeat-watcher.test.ts` — 7 cases (start/dispose, dead detection alert once, transition healthy→dead emits once, transition dead→healthy resets, multiple runs isolated, mock clock, consecutive 3 ticks → deadletter trigger).
-
-#### 9.1.B Retry executor (1.5 dev-days)
-
-**File mới:** `src/runtime/retry-executor.ts`
-
-```ts
-export interface RetryPolicy {
-	maxAttempts: number;        // default 3 (D6)
-	backoffMs: number;          // default 1000
-	jitterRatio: number;        // default 0.3 (±30%)
-	exponentialFactor: number;  // default 2
-	retryableErrors?: string[]; // glob patterns; empty = all retryable
-}
-
-export const DEFAULT_RETRY_POLICY: RetryPolicy = { maxAttempts: 3, backoffMs: 1000, jitterRatio: 0.3, exponentialFactor: 2 };
-
-export async function executeWithRetry<T>(
-	fn: (attempt: number) => Promise<T>,
-	policy: RetryPolicy = DEFAULT_RETRY_POLICY,
-	hooks?: { onAttemptFailed?: (attempt: number, error: Error, nextDelayMs: number) => void; onRetryGivenUp?: (attempts: number, error: Error) => void; signal?: AbortSignal }
-): Promise<T> { /* exponential backoff with jitter */ }
-
-function calculateDelay(attempt: number, policy: RetryPolicy): number {
-	const base = policy.backoffMs * Math.pow(policy.exponentialFactor, attempt - 1);
-	const jitter = (Math.random() * 2 - 1) * policy.jitterRatio * base;
-	return Math.max(0, base + jitter);
-}
-```
-
-**Wire vào `executeTeamRun`** opt-in (D5 + D19 state-machine semantics):
-- Read `loadConfig.config.reliability?.autoRetry` (default `false`, D5).
-- Nếu true → wrap `runTeamTask(task)` với `executeWithRetry`.
-- **State machine rules (D19):**
-  - Mỗi attempt → push entry `{ startedAt, endedAt, error? }` vào `task.attempts: Array<...>` (new field — schema additive).
-  - Task KHÔNG transition `running → failed → running` giữa các attempt (vi phạm monotonicity); thay vào đó, attempt N fail → đợi backoff → attempt N+1 vẫn `status="running"`, chỉ `attempts[]` mọc.
-  - Task transition `failed` CHỈ KHI maxAttempts exhausted; `task.error` reflect last error; artifact final chỉ finalize trên terminal attempt (không over-write per attempt).
-  - Idempotency requirement (risk Med-High): document trong release notes — `runTeamTask` phải idempotent hoặc user accept double-execute risk.
-- Mỗi attempt → emit `crew.task.retry_attempt{runId,taskId,attempt}` Counter, `crew.task.retry_delay_ms{runId,taskId}` Histogram observe.
-- Cuối cùng → record `crew.task.retry_count{runId,team}` Histogram observe (final attempt count).
-
-**Schema update `src/schema/config-schema.ts`:**
-```ts
-reliability: Type.Optional(Type.Object({
-	autoRetry: Type.Optional(Type.Boolean()),  // default false
-	retryPolicy: Type.Optional(Type.Object({
-		maxAttempts: Type.Optional(Type.Integer({ minimum: 1, maximum: 10 })),
-		backoffMs: Type.Optional(Type.Integer({ minimum: 100, maximum: 60000 })),
-		jitterRatio: Type.Optional(Type.Number({ minimum: 0, maximum: 1 })),
-		exponentialFactor: Type.Optional(Type.Number({ minimum: 1, maximum: 5 })),
-		retryableErrors: Type.Optional(Type.Array(Type.String())),
-	})),
-	autoRecover: Type.Optional(Type.Boolean()),  // default false
-	deadletterThreshold: Type.Optional(Type.Integer({ minimum: 1 })),  // default 3
-})),
-```
-
-**Tests:** `test/unit/retry-executor.test.ts` — 10 cases (success first try, fail then succeed, max attempts exhausted, abort signal, jitter range, retryable filter, custom policy override, mock clock backoff, hook callback fires).
-
-#### 9.1.C Crash recovery (1.5 dev-days)
-
-**File mới:** `src/runtime/crash-recovery.ts`
-
-**Logic:** session_start phát hiện run với status `running` từ session trước, **chỉ trigger recovery nếu thoả combinator (D20):**
-- `(manifest.status === "running")`
-- AND `(manifest.async?.pid === undefined OR pidIsDead(manifest.async.pid))` — reuse existing async.pid liveness check trong `src/extension/session-summary.ts`
-- AND `(no heartbeat OR isWorkerHeartbeatStale(heartbeat, deadMs) === true)` — reuse `isWorkerHeartbeatStale()` từ `src/runtime/worker-heartbeat.ts`
-
-Khi triggered:
-1. Read event-log cursor via `scanSequence(eventsPath)` từ `src/state/event-log.ts` (Phase 6 helper) — tìm last completed event seq.
-2. Compute "stale work":
-   - Tasks `running` nhưng heartbeat dead → mark `pending-recovery`.
-   - Tasks `completed`/`cancelled`/`failed` → preserve.
-3. NotificationRouter prompt: `"Run X was interrupted. Resume from event N? (Y/N)"` (D7) qua Phase 8 ConfirmOverlay.
-4. User confirm → reset stale tasks to `queued`, write resume event với metadata `{ recoveredFromSeq: N }`, emit `crew.run.resumed{runId, fromEventSeq}`.
-5. User decline → mark run `cancelled` với reason `"interrupted-not-resumed"`.
-
-**Skeleton:**
-
-```ts
-export interface RecoveryPlan {
-	runId: string;
-	resumableTasks: string[];   // taskIds to reset to queued
-	preservedTasks: string[];   // taskIds completed/cancelled (no change)
-	lastEventSeq: number;
-}
-
-export function detectInterruptedRuns(cwd: string, manifestCache: ManifestCache): RecoveryPlan[] { /* ... */ }
-export async function applyRecoveryPlan(plan: RecoveryPlan, ctx: ExtensionContext, registry: MetricRegistry): Promise<void> { /* ... */ }
-```
-
-**Wire vào `register.ts:session_start`:**
-```ts
-if (loadedConfig.config.reliability?.autoRecover === true) {
-	const plans = detectInterruptedRuns(ctx.cwd, manifestCache);
-	for (const plan of plans) {
-		// Use NotificationRouter + ConfirmOverlay prompt
-		notificationRouter.enqueue({
-			severity: "warning",
-			source: "crash-recovery",
-			runId: plan.runId,
-			title: `Run ${plan.runId} was interrupted`,
-			body: `${plan.resumableTasks.length} tasks pending recovery. Open dashboard → confirm to resume.`,
-			id: `recovery_prompt_${plan.runId}`,
-		});
-	}
-}
-```
-
-**Tests:** `test/integration/crash-recovery.test.ts` — 5 cases (no interrupted runs, single run resume, decline marks cancelled, multiple runs, completed tasks preserved).
-
-#### 9.1.D Deadletter queue (0.5 dev-day)
-
-**File mới:** `src/runtime/deadletter.ts`
-
-**Logic (D22 — 3 separate trigger paths):**
-- **Path (a) — retry exhaust:** trong `executeWithRetry` hooks `onRetryGivenUp(attempts, error)` → call `appendDeadletter({ reason: "max-retries", attempts, lastError })`.
-- **Path (b) — heartbeat watcher consecutive dead:** `HeartbeatWatcher.onDeadletterTrigger(runId, taskId)` (count = 3 ticks consecutive — xem 9.1.A) → call `appendDeadletter({ reason: "heartbeat-dead", attempts: 0 })`.
-- **Path (c) — threshold alert (separate from entry write):** Counter `crew.task.deadletter_total` rate > 3/hour (TimeWindowedCounter from 9.2.B) → NotificationRouter alert severity `error` với id `deadletter_threshold_${runId}` (dedup window 1h).
-
-Tất cả 3 paths đều:
-1. Append vào `<crewRoot>/state/runs/{runId}/deadletter.jsonl`.
-2. Emit `crew.task.deadletter{runId,taskId,reason}` Counter inc.
-
-```ts
-export interface DeadletterEntry {
-	taskId: string;
-	runId: string;
-	reason: "max-retries" | "heartbeat-dead" | "manual";
-	attempts: number;
-	lastError?: string;
-	timestamp: string;
-}
-
-export function appendDeadletter(manifest: TeamRunManifest, entry: DeadletterEntry): void { /* JSONL append */ }
-export function readDeadletter(manifest: TeamRunManifest): DeadletterEntry[] { /* read all */ }
-```
-
-**Tests:** `test/unit/deadletter.test.ts` — 4 cases (append, read, threshold trigger, persistence cross-session).
-
----
-
-### Phase 9.2 — Telemetry Pipeline (4 dev-days)
-
-#### 9.2.A Event-to-metric subscriber (1 dev-day)
-
-**File mới:** `src/observability/event-to-metric.ts`
-
-**Hardcoded mapping (D12):**
-
-```ts
-export function wireEventToMetrics(events: ExtensionAPI["events"], registry: MetricRegistry): { dispose: () => void } {
-	// Counters
-	const runCount = registry.registerCounter("crew.run.count", "Total runs by status");
-	const taskCount = registry.registerCounter("crew.task.count", "Total tasks by status");
-	const subagentCount = registry.registerCounter("crew.subagent.count", "Total subagent records by status");
-	const mailboxCount = registry.registerCounter("crew.mailbox.count", "Total mailbox messages by direction");
-	const deadletterCount = registry.registerCounter("crew.task.deadletter_total", "Deadletter triggers by reason");
-
-	// Gauges
-	const heartbeatStaleness = registry.registerGauge("crew.heartbeat.staleness_ms", "Heartbeat elapsed since last seen, milliseconds");
-
-	// Histograms
-	const runDuration = registry.registerHistogram("crew.run.duration_ms", "Run end-to-end duration, milliseconds");
-	const taskDuration = registry.registerHistogram("crew.task.duration_ms", "Task duration, milliseconds");
-	const retryCount = registry.registerHistogram("crew.task.retry_count", "Retries per task", [0, 1, 2, 3, 5, 10]);
-	const tokenUsage = registry.registerHistogram("crew.task.tokens_total", "Token usage per task");
-
-	const handlers: Array<[string, (data: any) => void]> = [
-		["crew.run.completed", (d) => { runCount.inc({ status: "completed" }); runDuration.observe({ team: d.team ?? "unknown" }, d.durationMs ?? 0); }],
-		["crew.run.failed", (d) => { runCount.inc({ status: "failed" }); }],
-		["crew.run.cancelled", (d) => { runCount.inc({ status: "cancelled" }); }],
-		["crew.subagent.completed", (d) => { subagentCount.inc({ status: d.status }); }],
-		["crew.mailbox.message", (d) => { mailboxCount.inc({ direction: d.direction }); }],
-		// ... etc
-	];
-
-	// D18: events.on() returns unsubscribe fn (EventBus interface). NO events.off() exists.
-	const unsubscribers: Array<() => void> = [];
-	for (const [event, handler] of handlers) {
-		const unsub = events?.on?.(event, handler);
-		if (unsub) unsubscribers.push(unsub);
-	}
-	return { dispose: () => { for (const unsub of unsubscribers) unsub(); unsubscribers.length = 0; } };
-}
-```
-
-**Tests:** `test/unit/event-to-metric.test.ts` — 8 cases (each event handler increments correct metric, dispose calls each unsubscribe fn, no-op nếu events undefined, dispose idempotent — calling 2x không crash, multiple subscribers parallel isolated, handler exception không break other handlers via EventBus safe wrapper).
-
-#### 9.2.B Metric retention (1 dev-day)
-
-**File mới:** `src/observability/metric-retention.ts`
-
-**Logic:** Streaming window 1h (D8) — mỗi metric value có timestamp; periodically (every 60s) → purge values older than window. Daily summary aggregation roll up vào persistent JSONL (9.2.D).
-
-```ts
-export class TimeWindowedCounter {
-	private events: { timestamp: number; labels: MetricLabels; delta: number }[] = [];
-	constructor(private windowMs: number = 3_600_000) {}
-	inc(labels: MetricLabels, delta = 1): void { /* push, then prune */ }
-	rate(labels: MetricLabels, durationMs: number): number { /* count events in last durationMs / durationMs */ }
-}
-```
-
-**Wire MetricRegistry:** option `retentionMs` per metric — default 1h cho counter rate; gauge giữ latest value (no retention); histogram observations retain all (memory bounded by labels cardinality).
-
-**Tests:** `test/unit/metric-retention.test.ts` — 5 cases (retain within window, prune outside, rate calculation, multiple labels isolated, mock clock).
-
-#### 9.2.C Histogram quantile (1 dev-day)
-
-**Update `metrics-primitives.ts`:** thêm method `quantile()`:
-
-```ts
-quantile(labels: MetricLabels, q: number): number {
-	const obs = this.observations.get(labelKey(labels));
-	if (!obs || obs.count === 0) return NaN;
-	const targetIdx = q * obs.count;
-	let cumulative = 0;
-	for (let i = 0; i < this.buckets.length; i++) {
-		cumulative += obs.counts[i];
-		if (cumulative >= targetIdx) {
-			const prevCum = cumulative - obs.counts[i];
-			const lower = i === 0 ? 0 : this.buckets[i - 1];
-			const upper = this.buckets[i];
-			// Linear interpolation within bucket
-			const fraction = (targetIdx - prevCum) / Math.max(1, obs.counts[i]);
-			return lower + fraction * (upper - lower);
-		}
-	}
-	return this.buckets[this.buckets.length - 1];  // overflow bucket
-}
-```
-
-**Tests:** `test/unit/metrics-primitives.test.ts` mở rộng — quantile p50/p95/p99 với fixture data; edge empty, edge single value, edge all in one bucket.
-
-#### 9.2.D Metric file sink (1 dev-day)
-
-**File mới:** `src/observability/metric-sink.ts`
-
-**Logic:** Tương tự Phase 8 `notification-sink.ts` — daily JSONL rotation, retention configurable. Sink writer chạy interval (default 60s) → snapshot registry → append. Reuse `redactSecrets` từ `diagnostic-export.ts` cho label values (precaution với secret patterns).
-
-```ts
-import { redactSecrets } from "../runtime/diagnostic-export.ts";  // Phase 8 helper
-import { logInternalError } from "../utils/internal-error.ts";
-
-export interface MetricSink {
-	writeSnapshot(snapshots: MetricSnapshot[]): void;
-	dispose(): void;
-}
-
-export interface MetricFileSinkOptions {
-	crewRoot: string;
-	registry: MetricRegistry;
-	retentionDays?: number;       // default 7
-	intervalMs?: number;           // default 60_000
-}
-
-export function createMetricFileSink(opts: MetricFileSinkOptions): MetricSink {
-	const dir = path.join(opts.crewRoot, "state", "metrics");
-	const retentionDays = opts.retentionDays ?? 7;
-	const writeSnapshot = (snapshots: MetricSnapshot[]): void => {
-		try {
-			const date = new Date().toISOString().slice(0, 10);
-			rotateOldFiles(dir, retentionDays);
-			fs.mkdirSync(dir, { recursive: true });
-			const redacted = redactSecrets(snapshots);
-			fs.appendFileSync(path.join(dir, `${date}.jsonl`), `${JSON.stringify({ exportedAt: new Date().toISOString(), snapshots: redacted })}\n`, "utf-8");
-		} catch (e) { logInternalError("metric-sink.write", e); }
-	};
-	const timer = setInterval(() => writeSnapshot(opts.registry.snapshot()), opts.intervalMs ?? 60_000);
-	return { writeSnapshot, dispose: () => clearInterval(timer) };
-}
-```
-
-**Tests:** `test/unit/metric-sink.test.ts` — 5 cases (write basic, daily rotation, retention prune, telemetry disabled no-op when not instantiated, dispose stops timer + secret redaction in labels).
-
----
-
-### Phase 9.3 — Export Adapters (3 dev-days)
-
-#### 9.3.A Prometheus exposition format (1 dev-day)
-
-**File mới:** `src/observability/exporters/prometheus-exporter.ts`
-
-```ts
-export function formatPrometheus(snapshots: MetricSnapshot[]): string {
-	const lines: string[] = [];
-	for (const snap of snapshots) {
-		lines.push(`# HELP ${snap.name} ${snap.description ?? ""}`);
-		lines.push(`# TYPE ${snap.name} ${snap.type}`);
-		// Format values per type with labels: name{label="value"} value timestamp
-		// ...
-	}
-	return lines.join("\n") + "\n";
-}
-```
-
-**Optional HTTP endpoint:** `team metrics --serve --port 9091` command starts simple `http.createServer` exposing `/metrics` endpoint. Off by default.
-
-**Tests:** `test/unit/prometheus-exporter.test.ts` — 6 cases (counter format, gauge format, histogram format with buckets, labels escaping, empty registry, special chars).
-
-#### 9.3.B OTLP HTTP exporter (1.5 dev-days, OPTIONAL — disable mặc định D10)
-
-**File mới:** `src/observability/exporters/otlp-exporter.ts`
-
-**Logic:** Convert MetricSnapshot → OTLP JSON format (HTTP/protobuf alt); POST đến endpoint config. Buffer batch 60s.
-
-```ts
-export interface OTLPExporterOptions {
-	endpoint: string;       // e.g. http://collector:4318/v1/metrics
-	headers?: Record<string, string>;
-	intervalMs?: number;    // default 60_000
-	timeoutMs?: number;     // default 10_000
-}
-
-export class OTLPExporter {
-	constructor(private opts: OTLPExporterOptions, private registry: MetricRegistry) {}
-	start(): void { /* setInterval push */ }
-	private async push(): Promise<void> {
-		const otlp = convertToOTLP(this.registry.snapshot());
-		try {
-			await fetch(this.opts.endpoint, { method: "POST", headers: { "content-type": "application/json", ...this.opts.headers }, body: JSON.stringify(otlp), signal: AbortSignal.timeout(this.opts.timeoutMs ?? 10_000) });
-		} catch (e) { logInternalError("otlp-export", e); }
-	}
-	dispose(): void { /* clearInterval */ }
-}
-
-function convertToOTLP(snapshots: MetricSnapshot[]): unknown { /* OpenTelemetry JSON spec */ }
-```
-
-**Schema config:**
-```ts
-otlp: Type.Optional(Type.Object({
-	enabled: Type.Optional(Type.Boolean()),
-	endpoint: Type.String(),
-	headers: Type.Optional(Type.Record(Type.String(), Type.String())),
-	intervalMs: Type.Optional(Type.Integer({ minimum: 5000 })),
-})),
-```
-
-**Tests:** `test/unit/otlp-exporter.test.ts` — 5 cases (format conversion, push success mock fetch, push timeout, dispose stops, disabled no-op).
-
-#### 9.3.C Adapter abstraction (0.5 dev-day)
-
-**File mới:** `src/observability/exporters/adapter.ts`
-
-```ts
-export interface MetricExporter {
-	name: string;
-	push(snapshots: MetricSnapshot[]): Promise<void>;
-	dispose(): void;
-}
-
-export class CompositeExporter implements MetricExporter {
-	name = "composite";
-	constructor(private exporters: MetricExporter[]) {}
-	async push(snapshots: MetricSnapshot[]): Promise<void> {
-		await Promise.allSettled(this.exporters.map((e) => e.push(snapshots)));
-	}
-	dispose(): void { for (const e of this.exporters) e.dispose(); }
-}
-```
-
-**Tests:** `test/unit/composite-exporter.test.ts` — 3 cases (push parallel, dispose all, error in one doesn't break others).
-
----
-
-### Phase 9.4 — UI & Commands (3 dev-days)
-
-#### 9.4.A `team metrics` command (1 dev-day)
-
-**Update `src/extension/team-tool/api.ts`:** thêm operation `metrics-snapshot`:
-
-```ts
-if (operation === "metrics-snapshot") {
-	const filter = typeof cfg.filter === "string" ? cfg.filter : undefined;  // glob pattern
-	const snapshots = getMetricRegistry().snapshot();
-	const filtered = filter ? snapshots.filter((s) => globMatch(s.name, filter)) : snapshots;
-	return result(JSON.stringify(filtered, null, 2), { action: "api", status: "ok" });
-}
-```
-
-**Slash command:** `/team-metrics [filter]` → wraps API call, prints formatted output.
-
-**Tests:** `test/unit/team-tool-metrics.test.ts` — 3 cases (snapshot all, filter glob, empty registry).
-
-#### 9.4.B Metrics dashboard pane (1 dev-day)
-
-**File mới:** `src/ui/dashboard-panes/metrics-pane.ts`
-
-**Render:** top 10 metrics by value, sparkline cho histogram p95 trend (last 60min stored in retention store).
-
-```ts
-export interface MetricsPaneOptions {
-	registry: MetricRegistry;
-	maxCounters?: number;  // default 10
-}
-
-// Signature consistent với Phase 8 panes — `(snapshot, opts?)`
-export function renderMetricsPane(snapshot: RunUiSnapshot | undefined, opts: MetricsPaneOptions): string[] {
-	if (!snapshot) return ["Metrics pane: snapshot unavailable"];
-	const metrics = opts.registry.snapshot();
-	const counters = metrics.filter((m) => m.type === "counter").slice(0, opts.maxCounters ?? 10);
-	const lines: string[] = ["Metrics top 10 counters:"];
-	for (const c of counters) {
-		// Format: name{labels}: value
-		// ...
-	}
-	return lines;
-}
-```
-
-**Update `src/ui/run-dashboard.ts`:** key `6` → `activePane = "metrics"`; help line update; constructor receives `registry` reference qua `RunDashboardOptions`.
-
-**Tests:** `test/unit/metrics-pane.test.ts` — 4 cases.
-
-#### 9.4.C Diagnostic export include metrics (0.5 dev-day) — **Schema version bump (D21)**
-
-**Update `src/runtime/diagnostic-export.ts`** (Phase 8 file, 4303 bytes — verified):
-
-```ts
-// Schema additive — backward-compat for consumers reading old DiagnosticReport
-export interface DiagnosticReport {
-	schemaVersion?: number;             // NEW v2 — undefined treated as v1
-	runId: string;
-	exportedAt: string;
-	manifest: TeamRunManifest;
-	tasks: TeamTaskState[];
-	recentEvents: TeamEvent[];
-	heartbeat: HeartbeatSummary;
-	agents: unknown[];
-	envRedacted: Record<string, string>;
-	metricsSnapshot?: MetricSnapshot[]; // NEW — optional, only set when registry available
-}
-
-// In exportDiagnostic(): apply redactSecrets() recursive on metricsSnapshot label values
-// before writing — secret patterns (token/key/password/secret/credential/auth) có thể xuất hiện
-// trong label values hoặc histogram metadata.
-```
-
-**Caller (commands.ts handler):** pass per-session `MetricRegistry` reference vào `exportDiagnostic(ctx, runId, { registry })`. Nếu registry undefined (telemetry disabled hoặc Phase 9 chưa wired), field `metricsSnapshot` để undefined → backward-compat with Phase 8 consumer.
-
-**Tests:** `test/unit/diagnostic-export.test.ts` extend — 2 cases:
-1. Verify `metricsSnapshot` included khi registry passed; `schemaVersion === 2`.
-2. Verify secret labels redacted (e.g., metric `crew.api.key_calls{auth_token="abc"}` → `auth_token: "***"`).
-
----
-
-### Phase 9.5 — Wiring & Tests (3 dev-days)
-
-#### 9.5.A Wire register.ts (1 dev-day) — **Per-session pattern (D17)**
-
-**Update `src/extension/register.ts`:**
-```ts
-import { createMetricRegistry } from "../observability/metric-registry.ts";  // factory, not singleton
-import { wireEventToMetrics } from "../observability/event-to-metric.ts";
-import { HeartbeatWatcher } from "../runtime/heartbeat-watcher.ts";
-import { detectInterruptedRuns } from "../runtime/crash-recovery.ts";
-import { createMetricFileSink } from "../observability/metric-sink.ts";
-
-// Module-scope state cho session (consistent với notificationRouter pattern Phase 8):
-let metricRegistry: MetricRegistry | undefined;
-let eventMetricSub: { dispose: () => void } | undefined;
-let metricSink: MetricSink | undefined;
-let heartbeatWatcher: HeartbeatWatcher | undefined;
-
-const configureObservability = (ctx: ExtensionContext): void => {
-	// Dispose existing per-session resources first (idempotent)
-	heartbeatWatcher?.dispose();
-	metricSink?.dispose();
-	eventMetricSub?.dispose();
-	metricRegistry?.dispose();
-
-	const config = loadConfig(ctx.cwd).config;
-	if (config.observability?.enabled === false) {
-		metricRegistry = undefined; eventMetricSub = undefined; metricSink = undefined; heartbeatWatcher = undefined;
-		return;
-	}
-
-	metricRegistry = createMetricRegistry();
-	eventMetricSub = wireEventToMetrics(pi.events, metricRegistry);
-	if (config.telemetry?.enabled !== false) {
-		metricSink = createMetricFileSink({ crewRoot: projectCrewRoot(ctx.cwd), registry: metricRegistry, retentionDays: config.observability?.metricRetentionDays ?? 7 });
-	}
-	heartbeatWatcher = new HeartbeatWatcher({
-		cwd: ctx.cwd,
-		pollIntervalMs: config.observability?.pollIntervalMs ?? 5000,
-		manifestCache: getManifestCache(ctx.cwd),
-		registry: metricRegistry,
-		router: notificationRouter!,  // Phase 8 router required
-		onDeadletterTrigger: (runId, taskId) => {
-			// Path (b) D22 — call deadletter writer
-			appendDeadletter(loadRunManifestById(ctx.cwd, runId)!.manifest, { taskId, runId, reason: "heartbeat-dead", attempts: 0, timestamp: new Date().toISOString() });
-		},
-	});
-	heartbeatWatcher.start();
-
-	if (config.reliability?.autoRecover === true) {
-		const plans = detectInterruptedRuns(ctx.cwd, getManifestCache(ctx.cwd));
-		for (const plan of plans) {
-			notificationRouter?.enqueue({ id: `recovery_prompt_${plan.runId}`, severity: "warning", source: "crash-recovery", runId: plan.runId, title: `Run ${plan.runId} was interrupted`, body: `${plan.resumableTasks.length} tasks pending recovery. Open dashboard → confirm to resume.` });
-		}
-	}
-};
-
-// session_start hook:
-pi.on("session_start", (ctx) => {
-	currentCtx = ctx;
-	configureNotifications(ctx);     // Phase 8
-	configureObservability(ctx);      // Phase 9 NEW
-	// ... rest
-});
-
-// session_shutdown hook (extends Phase 8 cleanupRuntime):
-pi.on("session_shutdown", () => {
-	// Phase 9 cleanup (per-session, in reverse setup order)
-	heartbeatWatcher?.dispose();      heartbeatWatcher = undefined;
-	metricSink?.dispose();            metricSink = undefined;
-	eventMetricSub?.dispose();        eventMetricSub = undefined;
-	metricRegistry?.dispose();        metricRegistry = undefined;
-	// Phase 8 cleanup
-	notificationRouter?.dispose();
-	notificationSink?.dispose();
-	// ...
-});
-```
-
-**Wrap executeTeamRun với correlation (9.0.C):**
-```ts
-const traceId = newSpanId(runId);  // {runId}:main:1 from spanCounter
-withCorrelation({ traceId, spanId: traceId }, async () => {
-	await executeTeamRun(...);
-});
-```
-
-**Pass `registry` reference downstream:**
-- `metricRegistry` exposed qua `RegisterTeamCommandsDeps` interface (commands.ts) cho dashboard pane + diagnostic export.
-- `dispatchDiagnosticExport(ctx, runId, { registry: metricRegistry })` để 9.4.C có thể inject metrics snapshot.
-
-#### 9.5.B Tests + smoke (2 dev-days)
-
-**Unit (mới ~70 cases):**
-- metrics-primitives.test.ts (12)
-- metric-registry.test.ts (6)
-- correlation.test.ts (5)
-- heartbeat-gradient.test.ts (8)
-- heartbeat-watcher.test.ts (6)
-- retry-executor.test.ts (10)
-- deadletter.test.ts (4)
-- event-to-metric.test.ts (8)
-- metric-retention.test.ts (5)
-- metric-sink.test.ts (5)
-- prometheus-exporter.test.ts (6)
-- otlp-exporter.test.ts (5)
-- composite-exporter.test.ts (3)
-- team-tool-metrics.test.ts (3)
-- metrics-pane.test.ts (4)
-
-**Integration (mới ~7 cases):**
-- `crash-recovery.test.ts` — 5 sub-cases.
-- `retry-executor-roundtrip.test.ts` — task fail 2x, succeed 3rd → metric counter records 3 attempts.
-- `heartbeat-watcher-deadletter.test.ts` — 3 dead detections in 1h → deadletter triggered + alert.
-- `metric-pipeline-end-to-end.test.ts` — emit events → snapshot via team-metrics → values match.
-- `correlation-cross-component.test.ts` — start run → subagent spawn → mailbox event — all events share traceId.
-- `prometheus-export.test.ts` — start run, fetch /metrics endpoint, verify format.
-- `otlp-export-mock.test.ts` — mock collector, verify POST body schema.
-
-**Smoke manual (10 scenarios):**
-1. Run team, finish → `/team-metrics` shows `crew.run.count{status=completed}=1`.
-2. Filter: `/team-metrics crew.task.*` shows only task metrics.
-3. Set `reliability.autoRetry=true`, fail task 2x → metric `retry_count` shows 3 attempts.
-4. Kill foreground process mid-run → reopen session → confirm prompt → resume → tasks continue.
-5. Set `reliability.autoRecover=false` → kill process → reopen → no prompt → run cancelled.
-6. Heartbeat stuck > 5min → notification toast → metric `heartbeat.dead_total` inc.
-7. Trigger 4 deadletter messages → alert toast severity error.
-8. `<crewRoot>/state/metrics/{date}.jsonl` populated after 60s.
-9. `/team-metrics` filter on Counter histogram quantile p95.
-10. OTLP export enabled with mock collector → verify push every 60s.
-
-## 3. Wave Organization
-
-```
-Wave 1 (sequential, 4 days) — Foundation must come first
-└─ 9.0 (.A → .B → .C → .D → .E preflight)
-
-Wave 2 (parallel, 5 days) — depends on Wave 1
-├─ 9.1.A Heartbeat watcher
-├─ 9.1.B Retry executor
-└─ 9.1.D Deadletter (depends on 9.1.B + 9.1.A)
-   ⤷ 9.1.C Crash recovery (depends on 9.0.C correlation)
-
-Wave 3 (parallel, 4 days) — depends on Wave 1
-├─ 9.2.A Event-to-metric subscriber
-├─ 9.2.B Metric retention
-├─ 9.2.C Histogram quantile (extends 9.0.A)
-└─ 9.2.D Metric sink
-
-Wave 4 (parallel, 3 days) — depends on Wave 3
-├─ 9.3.A Prometheus exporter
-├─ 9.3.B OTLP exporter (optional)
-├─ 9.3.C Adapter abstraction
-└─ 9.4.A team metrics command
-   ⤷ 9.4.B Metrics dashboard pane
-   ⤷ 9.4.C Diagnostic include metrics
-
-Wave 5 (sequential, 3 days)
-├─ 9.5.A Wire register.ts
-└─ 9.5.B Tests + smoke validation
-```
-
-**Total estimate: 19.5-22.5 dev-days** (Theme B+C combined; Wave 1 +0.5d for 9.0.E preflight).
-
-## 4. Files Affected
-
-### New (33 files — +1 cho 9.0.E preflight test)
-| Path | Purpose | Est LOC |
-|---|---|---|
-| `src/observability/metrics-primitives.ts` | Counter/Gauge/Histogram base | ~200 |
-| `src/observability/metric-registry.ts` | Singleton registry | ~120 |
-| `src/observability/correlation.ts` | AsyncLocalStorage context | ~80 |
-| `src/observability/event-to-metric.ts` | Event subscriber → metrics | ~150 |
-| `src/observability/metric-retention.ts` | Time-windowed counter | ~80 |
-| `src/observability/metric-sink.ts` | JSONL sink + rotation | ~100 |
-| `src/observability/exporters/prometheus-exporter.ts` | Prometheus format | ~120 |
-| `src/observability/exporters/otlp-exporter.ts` | OTLP HTTP exporter (optional) | ~180 |
-| `src/observability/exporters/adapter.ts` | Composite + interface | ~60 |
-| `src/runtime/heartbeat-gradient.ts` | Classifier function (uses `WorkerHeartbeatState`) | ~60 |
-| `src/runtime/heartbeat-watcher.ts` | Background poller (per-session, reuse loadRunManifestById + classifyHeartbeat) | ~170 |
-| `test/unit/extension-api-surface.test.ts` | **9.0.E preflight** — verify `events.on()` returns unsubscribe + `events.off` does NOT exist + `WorkerHeartbeatState` fields | ~110 |
-| `src/runtime/retry-executor.ts` | Backoff + jitter | ~120 |
-| `src/runtime/crash-recovery.ts` | Detect + apply plan | ~180 |
-| `src/runtime/deadletter.ts` | Append + read JSONL | ~80 |
-| `src/ui/dashboard-panes/metrics-pane.ts` | Metrics pane renderer | ~80 |
-| `test/unit/metrics-primitives.test.ts` | | ~250 |
-| `test/unit/metric-registry.test.ts` | | ~100 |
-| `test/unit/correlation.test.ts` | | ~120 |
-| `test/unit/heartbeat-gradient.test.ts` | | ~140 |
-| `test/unit/heartbeat-watcher.test.ts` | | ~170 |
-| `test/unit/retry-executor.test.ts` | | ~220 |
-| `test/unit/deadletter.test.ts` | | ~90 |
-| `test/unit/event-to-metric.test.ts` | | ~180 |
-| `test/unit/metric-retention.test.ts` | | ~110 |
-| `test/unit/metric-sink.test.ts` | | ~120 |
-| `test/unit/prometheus-exporter.test.ts` | | ~150 |
-| `test/unit/otlp-exporter.test.ts` | | ~140 |
-| `test/unit/composite-exporter.test.ts` | | ~80 |
-| `test/unit/team-tool-metrics.test.ts` | | ~80 |
-| `test/unit/metrics-pane.test.ts` | | ~80 |
-| `test/integration/crash-recovery.test.ts` | | ~200 |
-| `test/integration/retry-executor-roundtrip.test.ts` | | ~150 |
-| `test/integration/heartbeat-watcher-deadletter.test.ts` | | ~150 |
-| `test/integration/metric-pipeline-end-to-end.test.ts` | | ~180 |
-| `test/integration/correlation-cross-component.test.ts` | | ~150 |
-| `test/integration/prometheus-export.test.ts` | | ~120 |
-| `test/integration/otlp-export-mock.test.ts` | | ~140 |
-
-### Modified (10 files)
-| Path | Change |
-|---|---|
-| `src/extension/register.ts` | Wire registry, event-metric subscriber, heartbeat watcher, retry/recovery, OTLP exporter |
-| `src/extension/team-tool/api.ts` | Thêm operation `metrics-snapshot` |
-| `src/extension/registration/commands.ts` | Slash command `/team-metrics`; recovery confirm flow |
-| `src/runtime/team-runner.ts` | Optional `executeWithRetry` wrap khi `autoRetry=true` |
-| `src/runtime/task-runner.ts` | Emit retry attempt events; correlation context wrap |
-| `src/ui/heartbeat-aggregator.ts` (Phase 8) | Switch internal classifier sang `heartbeat-gradient.ts`; emit metrics |
-| `src/ui/run-dashboard.ts` | Pane `6` metrics; help line |
-| `src/runtime/diagnostic-export.ts` (Phase 8) | Include `metricsSnapshot` field |
-| `src/schema/config-schema.ts` | Thêm `reliability` + `otlp` sections |
-| `src/config/{config.ts,defaults.ts}` | Parse + defaults |
-| `package.json` | Bump `0.1.34` → `0.1.35` |
-
-## 5. Risk Assessment
-
-| Risk | Likelihood | Impact | Mitigation |
-|---|---|---|---|
-| Correlation propagation chạm hầu hết module | High | Med | AsyncLocalStorage tự động — không phải pass thủ công; test isolation cross-async boundary |
-| `executeWithRetry` double-execute task on poorly-idempotent ops | Med | **High** | Default off (D5); D19 state-machine rules (no transition `failed → running`); user explicit opt-in; documentation warn idempotency requirement |
-| Crash recovery race với new run start cùng runId | Low | High | D20 combinator: status==="running" AND no async.pid alive AND heartbeat dead; reuse existing async.pid liveness check; recovery prompt blocking until user confirms |
-| Heartbeat watcher poll burns CPU | Low | Low | 5s default conservative; configurable; only iterate active runs (`status === "running"`) |
-| MetricRegistry memory leak với high-cardinality labels | Med | Med | Cap label count per metric (warn ở 1000); document anti-pattern |
-| OTLP export network failure spam logs | Low | Low | Swallow errors via `logInternalError`; circuit-breaker after 5 consecutive fails |
-| Histogram quantile inaccurate với fixed buckets | Med | Low | Document approximation; allow custom buckets per metric |
-| Background watcher leak nếu session_shutdown miss | Low | Med | Per-session pattern (D17) — dispose ordering tested in 9.5.B; idempotent dispose |
-| `events.jsonl` corruption blocks recovery | Low | High | Recovery validate seq monotonic via `scanSequence`; fallback "cancel run" if event log unreadable |
-| Metric sink file lock contention | Low | Low | `appendFileSync` synchronous within process; cross-process not supported (document) |
-| Retry policy over-aggressive → task storm | Med | Med | Default maxAttempts=3 conservative; jitter prevent thundering herd |
-| Deadletter false positive on transient errors | Med | Med | Threshold default 3 attempts; user override per task; deadletter reversible (manual reset) |
-| **`events.off` không tồn tại** trên ExtensionAPI EventBus | Mitigated | Was High | **D18**: 9.0.E preflight test verify; capture unsubscribe fn từ `events.on()` return — pattern matches existing `src/ui/render-scheduler.ts` |
-| **Naming mismatch `WorkerHeartbeat` vs actual `WorkerHeartbeatState`** | Mitigated | Was High | 9.0.E preflight test verify field names; explicit import từ `worker-heartbeat.ts` (NOT alias) |
-| **MetricRegistry singleton state leak across sessions** | Mitigated | Was Med | **D17**: per-session instance pattern; dispose trong session_shutdown |
-| **DiagnosticReport schema breaking** (extra `metricsSnapshot` field) | Mitigated | Was Med | **D21**: `schemaVersion: 2` bump; field optional (undefined for v1 readers); secret redaction recursive |
-| **Deadletter trigger ambiguity** (3 paths conflate) | Mitigated | Was Med | **D22**: 3 explicit trigger paths separated trong code (not one mega-handler) |
-| **Recovery race với existing async.pid liveness check** | Mitigated | Was High | **D20** combinator reuses existing logic; new path không override existing async.pid check |
-
-## 6. Testing Strategy
-
-**Unit-level (~70 cases):** xem mục 9.5.B chi tiết.
-
-**Integration (~7 scenarios):** xem mục 9.5.B.
-
-**Performance budget:**
-- Counter inc < 1μs.
-- Histogram observe < 5μs.
-- Registry snapshot full < 50ms cho 100 metrics.
-- Heartbeat watcher tick < 100ms cho 50 active runs.
-- Retry backoff jitter calculation < 1μs.
-- Crash recovery detection < 200ms cho 50 runs.
-
-**Property-based (optional):**
-- Histogram quantile monotonicity (q1 < q2 ⇒ result(q1) ≤ result(q2)).
-- Retry executor convergence (eventually success or give up within maxAttempts).
-
-**Smoke manual (10 scenarios):** xem mục 9.5.B.
-
-## 7. Open Questions (Pre-decide before Wave 1)
-
-| P | Câu hỏi | Default đề xuất | Tác động |
-|---|---|---|---|
-| **P1** | Correlation ID format? | `{runId}:{taskId}:{spanCounter}` (D3) | Human-readable, deterministic |
-| **P2** | Retry policy default config | `maxAttempts=3, backoffMs=1000, jitterRatio=0.3` (D6) | Industry standard |
-| **P3** | Crash recovery: auto-resume vs prompt? | **Prompt** via Phase 8 ConfirmOverlay (D7) | Avoid replay risk |
-| **P4** | Metric retention window default | 1h streaming, 24h JSONL (D8) | Cover 95% debug needs |
-| **P5** | Histogram bucket strategy | Fixed exponential (D2) | Simple, predictable |
-| **P6** | OTLP export priority | Implement, default-off (D10) | Enable team có observability stack |
-| **P7** | Deadletter threshold default | >3 messages/hour alert (D11) | Conservative, false-positive minimal |
-| **P8** | Background watcher polling interval | 5s default, 1-60s configurable (D9) | Balance responsiveness vs CPU |
-
-**All P1-P8 decisions defaulted in D-table (mục 1.B).** User có thể override qua config nhưng default sane.
-
-## 8. Dependencies & Sequencing
-
-```
-Phase 7 (DONE) ──► Phase 8 (Operator UX) ──► Phase 9 Wave 1 (Foundation)
-                              │                       │
-                              │              ┌────────┼────────┐
-                              ▼              ▼        ▼        ▼
-                   ConfirmOverlay reuse → 9.1 Reliability  9.2 Telemetry  9.3 Exporters
-                   NotificationRouter reuse        │              │              │
-                   diagnostic-export extend        └──────────────┼──────────────┘
-                                                                  ▼
-                                                            9.4 UI/Commands
-                                                                  │
-                                                                  ▼
-                                                            9.5 Wiring + Tests
-```
-
-**Hard prerequisites Phase 8:**
-- ✅ `NotificationRouter` (Phase 8.3.A) — used by 9.1.A/9.1.C/9.1.D for alerts.
-- ✅ `ConfirmOverlay` (Phase 8.0) — used by 9.1.C recovery prompt.
-- ✅ `diagnostic-export.ts` (Phase 8.2.D) — extended in 9.4.C.
-
-**Parallelization opportunity:** Wave 2 vs Wave 3 có thể chạy song song (chỉ share Wave 1 foundation).
-
-## 9. Effort Summary
-
-| Wave | Items | Dev-days | Parallelizable |
-|---|---|---|---|
-| 1 | 9.0.A → B → C → D → **E preflight** | 4 | No (sequential foundation; 9.0.E gates Wave 2) |
-| 2 | 9.1.A + 9.1.B + 9.1.C + 9.1.D | 5 | Partial (4 streams, .C depends .A/.B) |
-| 3 | 9.2.A + 9.2.B + 9.2.C + 9.2.D | 4 | Yes (4 streams, low overlap) |
-| 4 | 9.3.A + 9.3.B + 9.3.C + 9.4.A + 9.4.B + 9.4.C | 3 | Partial (5 streams; UI track 9.4.A→B→C critical path 2.5d) |
-| 5 | 9.5.A + 9.5.B | 3 | No |
-| **Total** | **19 sub-phases** | **19.5-22.5** | — |
-
-**So với Phase 8 (14-18 dev-days):** Phase 9 lớn hơn ~25%, risk cao hơn vì touches state machine.
-
-## 10. Acceptance Checklist (Wave 5 exit criteria)
-
-- [x] Tất cả checkbox 9.0 → 9.5 (bao gồm 9.0.E preflight) tick `[x]`.
-- [x] `npm test` pass: **389 unit** + **45 integration**, 0 fail (2026-04-29).
-- [x] `npm run typecheck` clean.
-- [x] Manual smoke 10 scenarios pass.
-- [x] Performance budget thỏa: counter 0.597µs, histogram 0.551µs, snapshot 0.159ms, heartbeat watcher 61.777ms/50 runs, recovery detect 27.036ms/50 runs.
-- [x] No regression: Phase 7+8 tests vẫn pass (full suite clean).
-- [x] Config breaking? **No.** Schema additive (`reliability`, `otlp`, `observability` sections optional).
-- [x] Default behavior unchanged: `autoRetry=false`, `autoRecover=false`, `otlp.enabled=false`, `observability.enabled` default `true` (sink/watcher gated bởi telemetry).
-- [ ] Bump package version for next release (current workspace remained on `0.1.35`; release not requested in this Phase 9 implementation turn).
-- [x] Migration guide trong README/release notes section.
-- [x] **D18 verified**: 0 `events.off?.` references in Phase 9 code; all subscriptions use returned unsubscribe fn.
-- [x] **D17 verified**: 0 module-level `globalRegistry`/singleton patterns; all observability state per-session, disposed in session_shutdown.
-- [x] **D21 verified**: DiagnosticReport schemaVersion=2 khi metricsSnapshot present; schemaVersion undefined cho Phase 8 reports.
-- [x] **No listener leak** test: 3x session_start/shutdown cycles → 0 residual subscriptions on `pi.events`.
-
-## 11. Out of Scope (defer Phase 10+)
-
-- Multi-host metric aggregation (cluster-wide registry).
-- Slack/Discord webhook adapter (router supports custom sink, not built-in).
-- t-digest histogram algorithm (defer; fixed buckets sufficient).
-- Tracing UI (only metrics + correlation propagation in 9; trace viewer Phase 10).
-- Auto-tuning retry policy (ML-based) — stay manual config Phase 9.
-- Metric drift detection / anomaly alert beyond simple threshold.
-- Custom event-to-metric mapping via DSL (hardcoded core only).
-- pprof profiling export.
-- Cross-language metric sharing (Pi-only Phase 9).
-
-## 12. Path X Roadmap Summary
-
-| Phase | Theme | Effort | Status |
-|---|---|---|---|
-| 6 | `.crew/` migration + autonomous policy | ~12d | ✅ DONE |
-| 7 | UI Optimization (snapshot cache + render scheduler + 4 panes) | ~18d | ✅ DONE |
-| **8** | **Operator Experience (Theme A)** | **14-18d** | ✅ **DONE** (verified 351 unit + 44 integration pass, version 0.1.34, all 17 sub-phases shipped) |
-| **9** | **Observability + Reliability (Theme B+C)** | **19.5-22.5d** | ✅ **IMPLEMENTED** (verified 389 unit + 45 integration pass in workspace) |
-| 10+ | TBD: Performance baseline (Theme D), distributed coordination, multi-host | — | Future |
-
-**Path X total to Phase 9 done: ~63-67 dev-days** (Phase 6+7+8 done = 44d; Phase 9 = 19.5-22.5d remaining).
-
-## 13. Implementation Kickoff Checklist (Pre-Wave 1)
-
-Trước khi bắt đầu Wave 1 Phase 9, verify:
-
-- [x] Phase 8 đã ship (`NotificationRouter`, `ConfirmOverlay`, `MailboxDetailOverlay/Compose/Preview/AgentPicker`, `heartbeat-aggregator.ts`, `health-pane.ts`, `diagnostic-export.ts`, `notification-sink.ts` available — verified existence + tests pass).
-- [x] `npm test` baseline pass (351 unit + 44 integration từ Phase 8 — verified 2026-04-29).
-- [x] `npm run typecheck` clean (verified Phase 8).
-- [x] P1-P8 defaults reviewed (mục 7) — đã default trong D-table.
-- [x] Branch mới skipped intentionally — user requested no separate branch.
-- [x] Read `src/state/event-log.ts` để hiểu sequence cursor pattern — confirmed `seq` metadata + `sequencePath()` + `scanSequence()` + `sequenceCache` infrastructure present.
-- [x] Read `src/runtime/worker-heartbeat.ts` để identify actual interface name — confirmed `WorkerHeartbeatState` (NOT "WorkerHeartbeat") + helper `isWorkerHeartbeatStale`.
-- [x] Read `src/runtime/diagnostic-export.ts` — confirmed Phase 8 file structure (`DiagnosticReport` interface + `redactSecrets` regex `/(token|key|password|secret|credential|auth)/i`).
-- [x] Verify ExtensionAPI surface — confirmed `EventBus.on()` returns unsubscribe fn (via `node_modules/@mariozechner/pi-coding-agent/dist/core/event-bus.d.ts`); **NO `events.off()` exists** → use returned unsubscribe (D18).
-- [x] Read `src/runtime/team-runner.ts:executeTeamRun` để identify correlation wrap point.
-- [x] Confirm Node.js >= 20 (AsyncLocalStorage stable since Node 16; package engines require Node >=20).
-- [x] Decide nếu OTLP export ship trong Phase 9 hay defer Phase 10 (shipped default-off per D10).
-- [x] **Wave 1 entry gate: 9.0.E preflight test pass** — block Wave 2 nếu fail.
-
-**Sẵn sàng triển khai Phase 9 Path X. Phase 8 verified DONE.**
-
----
-
-**Note on Theme B vs Theme C balance:** Phase 9 này combine 2 themes vì 5 synergy critical (mục 1.A). Nếu trong quá trình Wave 2/3 phát hiện effort blow up, có thể split:
-- Phase 9a = B only (Wave 1 + Wave 3 + 9.4.A/B + part 9.5) ~12.5 dev-days (incl. 9.0.E preflight).
-- Phase 9b = C only (Wave 1 reuse + Wave 2 + part 9.4.C + part 9.5) ~10 dev-days.
-
-Decision split chỉ đưa ra khi có data thực tế từ Wave 1 progress.
-
----
-
-## Appendix A — Review Fixes Applied (2026-04-29)
-
-Plan đã được update post-review với các blocking issues đã giải quyết:
-
-| Issue | Fix | Reference |
-|---|---|---|
-| `WorkerHeartbeat` vs actual `WorkerHeartbeatState` | Replace tất cả references; explicit import | 9.0.D, 9.1.A, D-decisions |
-| `events.off?.()` không tồn tại trên EventBus | Use `events.on()` returned unsubscribe fn pattern | 9.2.A, D18, 9.0.E preflight |
-| MetricRegistry singleton dispose semantics ambiguous | Per-session instance pattern (consistent Phase 8) | 9.0.B, 9.5.A, D17 |
-| 9.0.E preflight ExtensionAPI verify thiếu | Added new sub-phase + test file | 9.0.E (NEW) |
-| Retry executor state-machine semantics chưa rõ | Document attempts[] + no `failed → running` transition | 9.1.B, D19 |
-| Crash recovery race với async.pid liveness | Combinator clause uses existing logic | 9.1.C, D20 |
-| Deadletter trigger 3 paths conflate | Separate explicit paths (a/b/c) | 9.1.D, D22 |
-| DiagnosticReport schema breaking | schemaVersion: 2 + redactSecrets recursive | 9.4.C, D21 |
-| `renderMetricsPane` signature lệch Phase 8 pattern | Change to `(snapshot, opts: { registry })` | 9.4.B |
-| Naming convention regex redundant | Tighten `^crew\.[a-z]+\.[a-z][a-z_]*$` | 9.0.B, D13 |
-| 9.1.A `for (const task of /* loaded.tasks */)` placeholder | Resolved với `loadRunManifestById(...).tasks` | 9.1.A skeleton |
-| 9.5.A wire pseudocode `..., registry` placeholder | Spec rõ `MetricFileSinkOptions` interface | 9.2.D, 9.5.A |
-| Phase 8 status label "NEXT" nhưng đã DONE | Update Path X table → ✅ DONE | Section 12 |
-| Acceptance no-listener-leak test thiếu | Added 3x cycle test | Section 10 |