@jaggerxtrm/specialists 3.17.0 → 3.18.0

package/docs/nodes.md

@@ -0,0 +1,231 @@
+# Nodes: CLI-native coordinator, runtime architecture, and operator flow
+
+## Doc contract
+
+- **Source of truth:**
+  - `src/specialist/node-contract.ts`
+  - `src/specialist/node-supervisor.ts`
+  - `src/specialist/job-control.ts`
+  - `src/cli/node.ts`
+  - `src/specialist/observability-sqlite.ts`
+- **Drift rule:** update this doc in the same change whenever node contract, lifecycle, or `sp node` CLI behavior changes.
+
+---
+
+## 1) Architecture
+
+Node execution is split into three roles:
+
+- **Coordinator specialist (LOW permission):** CLI-native orchestrator.
+- **NodeSupervisor:** effect executor and lifecycle authority.
+- **Members:** worker specialists managed by NodeSupervisor.
+
+Coordinator decides *what to do next* and does so by calling `sp node` commands. NodeSupervisor executes, persists state, and enforces lifecycle semantics.
+
+---
+
+## 2) Coordinator behavior (CLI-native)
+
+The coordinator no longer emits a JSON orchestration schema as its control output.
+
+Instead it:
+1. Calls `sp node ... --json` commands,
+2. Reads structured responses,
+3. Issues follow-up commands,
+4. Uses `wait-phase` as the phase barrier,
+5. Calls `complete` when node goals are satisfied.
+
+### Required command set
+
+- `sp node spawn-member --node $NODE_ID --member-key <key> --specialist <name> [--bead <id>] [--phase <id>] [--json]`
+- `sp node create-bead --node $NODE_ID --title '...' [--type task] [--priority 2] [--depends-on <id>] [--json]`
+- `sp node complete --node $NODE_ID --strategy <pr|manual> [--json]`
+- `sp node wait-phase --node $NODE_ID --phase <id> --members <k1,k2,...> [--json]`
+- `sp ps` (or `sp ps --node <id>` when available) for node snapshot/status
+- `sp feed --node <id> [--json]` for node event stream
+- `sp steer <coordinator-job-id> "..."` for coordinator steering
+- `sp attach <coordinator-job-id>` for coordinator attach
+- `sp result <node-ref>:<member-key>` for member result retrieval
+
+Node refs accept any unique prefix (e.g. `research`, `research-5eaf`, or full ID).
+
+---
+
+## 3) CLI command reference
+
+| Command | Purpose | Typical usage |
+|---|---|---|
+| `sp ps` | Read node snapshot and job health | Before and after every orchestration step |
+| `sp node spawn-member` | Start a member for a phase/task | Launch explore/impl/review workers |
+| `sp node wait-phase` | Enforce phase barrier for declared members | Block transition until phase members finish |
+| `sp node create-bead` | Create tracked follow-up work | Persist discovered blockers/tasks |
+| `sp node complete` | Finalize node run with strategy | End run via `pr` or `manual` completion |
+
+---
+
+## 4) Wait-phase semantics
+
+`sp node wait-phase` is the canonical phase barrier.
+
+- Coordinator must provide the phase id and explicit member key set.
+- Progression to the next phase is blocked until wait-phase reports completion.
+- This applies equally to review/fix/re-review loops.
+
+Recommended pattern:
+1. spawn all members for phase N,
+2. call `wait-phase` for phase N,
+3. read `sp ps --node <id> --json`,
+4. decide next phase (or completion).
+
+---
+
+## 5) Error handling model
+
+Coordinator handles command errors by reading error JSON and adapting.
+
+Recovery loop:
+1. run command with `--json`,
+2. inspect error payload,
+3. correct arguments or sequencing,
+4. retry boundedly,
+5. if unrecoverable, track via `create-bead` and choose explicit completion strategy.
+
+Examples:
+- unknown member in `wait-phase` -> refresh via `status`, retry with valid members.
+- invalid spawn args -> correct `member-key` / `phase` / `specialist`, retry.
+- completion rejected by state -> satisfy unmet prerequisites, then retry `complete`.
+
+---
+
+## 6) Lifecycle and state machine
+
+Node states:
+- `created`
+- `starting`
+- `running`
+- `waiting`
+- `degraded`
+- `awaiting_merge`
+- `fixing_after_review`
+- `failed`
+- `error`
+- `done`
+- `stopped`
+
+`awaiting_merge` is terminal for the node runtime loop under PR-based completion.
+
+---
+
+## 7) Completion behavior
+
+`sp node complete --strategy <pr|manual>` decides publication semantics:
+
+- `pr`: run ends in `awaiting_merge` when completion intent succeeds.
+- `manual`: run ends in `done` when completion checks pass.
+
+Coordinator should call `complete` only after phase barriers and status checks indicate readiness.
+
+---
+
+## 7.5) `completion_strategy` in node configs
+
+The `completion_strategy` field in `.node.json` configs controls how NodeSupervisor handles coordinator completion:
+
+```json
+{
+  "completion_strategy": "manual"  // or "pr" (default)
+}
+```
+
+### Values
+
+| Strategy | Behavior | Use case |
+|---|---|---|
+| `pr` (default) | When coordinator job reaches `done` with valid output, node auto-closes to `done` state. | Implementation nodes — auto-publish work via PR |
+| `manual` | When coordinator job reaches `done` with valid output, node transitions to `waiting`. Operator must close via `sp node stop`. | Research/interactive nodes — operator reviews findings before closure |
+
+### Key behavior
+
+When the coordinator completes synthesis and its job status becomes `done`:
+
+1. NodeSupervisor checks `completion_strategy` from node config.
+2. If `manual`: node enters `waiting` state (coordinator_done_manual_completion transition).
+3. If `pr`: node enters `done` state directly (coordinator_done transition).
+
+This allows research nodes to persist in `waiting` after coordinator synthesis, giving the operator time to review findings, steer for more work, or close explicitly.
+
+### Example configs
+
+```json
+// config/nodes/research.node.json (interactive research)
+{
+  "completion_strategy": "manual",
+  "members": [...]
+}
+
+// Implementation nodes (auto-publish via PR)
+{
+  "completion_strategy": "pr",
+  "members": [...]
+}
+```
+
+---
+
+## 8) Member bootstrap behavior
+
+Members start in idle-wait bootstrap mode:
+- acknowledge readiness,
+- wait for coordinator steering,
+- avoid eager autonomous execution before routing.
+
+This is intentional and remains part of runtime safety behavior.
+
+---
+
+## 9) Observability and inspection
+
+Persisted SQLite surfaces:
+- `node_runs`
+- `node_members`
+- `node_events`
+- `node_memory`
+
+Useful CLI inspection commands:
+- `sp ps` (or `sp ps --node <id>` when available)
+- `sp feed --node <id> [--json]`
+- `sp steer <coordinator-job-id> "..."`
+- `sp attach <coordinator-job-id>`
+- `sp result <node-ref>:<member-key>`
+- `sp node members <id> [--json]`
+- `sp node memory <id> [--json]`
+
+Canonical dispatch event remains `action_written`.
+
+---
+
+## 10) Current `sp node` surface
+
+- `sp node run <config-or-name> [--inline JSON] [--bead <id>] [--context-depth <n>] [--json]`
+- `sp node list [--json]`
+- `sp node members <node-id> [--json]`
+- `sp node memory <node-id> [--json]`
+- `sp node stop <node-id> [--json]`
+- `sp node promote <node-id> <finding-id> --to-bead <bead-id> [--json]`
+
+Related top-level commands for node operations:
+- `sp ps` / `sp ps --node <id>`
+- `sp feed --node <id> [--json]`
+- `sp steer <coordinator-job-id> <message>`
+- `sp attach <coordinator-job-id>`
+- `sp result <node-ref>:<member-key>`
+
+---
+
+## 11) Practical coordinator heuristics
+
+- Re-check `sp ps --node <id> --json` before every phase transition.
+- Use `wait-phase` as mandatory barrier, not advisory.
+- Parallelize only disjoint mutating scopes.
+- Keep retries bounded and explicit.
+- Prefer explicit follow-up beads over silent failure when blocked.

package/docs/observability-metrics.md

@@ -0,0 +1,152 @@
+---
+title: RPC Observability Metrics Contract
+scope: observability-metrics
+category: reference
+version: 2.0.0
+updated: 2026-04-08
+synced_at: e7de0be7
+source_of_truth_for: "src/specialist/timeline-events.ts, src/pi/session.ts, src/cli/ps.ts, src/cli/result.ts"
+domain: ["observability", "metrics", "timeline"]
+summary: "Defines additive metrics from Pi RPC surfaced via specialists status/feed/timeline. v2.0 adds extension_error, model_change events; enriched compaction/retry fields; token_usage CLI display."
+---
+
+# RPC Observability Metrics Contract
+
+> **Bridge-era note (2026-06-02):** this file documents the current RPC/timeline metrics emitted by specialists. New Prometheus-facing metric design lives in `docs/telemetry/prometheus-projection-contract.md`, and forensic event shape lives in `docs/telemetry/forensic-event-contract.md`. Keep this file for source mapping/backward compatibility until the projection exporter lands.
+
+## Cross-repo telemetry contract
+
+This file is the specialists-owned metrics contract. It defines what the runtime
+emits for jobs, turns, token usage, tool calls, retries, compactions, model
+changes, extension errors, and related agent lifecycle events.
+
+The platform path is deliberately split:
+
+- `~/projects/mercury/infra/MONITORING.md` and `docs/AGENT_MONITORING.md` own scraping, storage, alerting, dashboard infrastructure, and future Terraform/IaC wiring.
+- `~/dev/gitboard/docs/xtrm-observability-prd.md` owns how these metrics become operator-console panels, links, and agent-authored dashboards.
+- `~/second-mind/1-projects/xtrm/research/agentops-telemetry-for-specialists.md` is the current research note for expanding this contract into AgentOps metrics.
+
+Do not rely on host/container metrics alone for specialist health. Required
+specialist-level signals include queue age, job state, tool-call error rate, MCP
+latency/error rate, model-change history, token/cost totals, retry/compaction
+frequency, worktree cleanup state, and durable result availability.
+
+
+This document defines additive, backward-compatible metrics captured from Pi RPC and surfaced through specialists status/feed/timeline. For KPI analysis recipes, see `.xtrm/skills/default/using-kpi/SKILL.md`.
+
+## Metric Source Map
+
+| Metric | RPC source | Capture file | Persisted to |
+|---|---|---|---|
+| `token_usage.*` | `assistantMessageEvent.done`, `turn_end`, `agent_end` usage-like payloads | `src/pi/session.ts` (`findTokenUsage`) | `status.json.metrics`, `events.jsonl` (`token_usage`), `run_complete.metrics` |
+| `finish_reason` | `stopReason` / `finishReason` from `assistantMessageEvent.done`, `turn_end`, `agent_end` | `src/pi/session.ts` (`findFinishReason`) | `status.json.metrics`, `events.jsonl` (`finish_reason`), `run_complete.metrics` |
+| `turns` | `turn_start` count | `src/pi/session.ts` | `status.json.metrics`, `run_complete.metrics` |
+| `tool_calls` | `tool_execution_start` count | `src/pi/session.ts` (+ supervisor reconciliation) | `status.json.metrics`, `run_complete.metrics` |
+| `auto_compactions` | `auto_compaction_end` count | `src/pi/session.ts` | `status.json.metrics`, `events.jsonl` (`compaction`), `run_complete.metrics` |
+| `auto_retries` | `auto_retry_end` count | `src/pi/session.ts` | `status.json.metrics`, `events.jsonl` (`retry`), `run_complete.metrics` |
+| `extension_error` | Extension error callbacks | `src/pi/session.ts` | `events.jsonl` (`extension_error`) |
+| `model_change` | Model change callbacks (`set_model`, `cycle_model`) | `src/pi/session.ts` | `events.jsonl` (`model_change`) |
+
+## Timeline Additions (Additive)
+
+### Event Types
+
+**Core metrics:**
+- `token_usage` — token consumption snapshot (input, output, cache, total, cost)
+- `finish_reason` — why the model stopped (stop, length, toolUse, etc.)
+- `turn_summary` — per-turn metadata with context health
+- `compaction` — context compaction lifecycle
+- `retry` — automatic retry lifecycle
+
+**New in v2.0:**
+- `extension_error` — extension failure events
+- `model_change` — model switch events (`set_model`, `cycle_model`)
+
+### Enriched Event Fields
+
+**Compaction events** (`auto_compaction_start`, `auto_compaction_end`):
+- `tokens_before` — context size before compaction
+- `summary` — what was compacted
+- `first_kept_entry_id` — first entry retained after compaction
+
+**Retry events** (`auto_retry_start`, `auto_retry_end`):
+- `attempt` — current attempt number
+- `max_attempts` — configured retry limit
+- `delay_ms` — backoff delay before retry
+- `error_message` — transient error that triggered retry
+
+Existing jobs without these events remain valid.
+
+## Surface Coverage
+
+### JSON APIs
+
+- `specialists feed --json`: includes `metrics` envelope from status + additive events.
+- `feed_specialist` tool: includes `metrics` from status.
+- `specialists status --json`: includes per-job `metrics`.
+- `specialist_status` tool: includes per-job `metrics`.
+
+### Human-Readable CLI (v2.0)
+
+**`sp ps` (list view):**
+- Total tokens displayed alongside elapsed time: `15s · 2500 tok`
+
+**`sp ps <id>` (inspect view):**
+- Dedicated `tokens` line with breakdown: `input=500 · output=2000 · total=2500`
+- Dedicated `cost_usd` line: `cost_usd=$0.0125`
+
+**`sp result` (human mode):**
+- Metrics footer appended to output:
+  ```
+  --- metrics: input=500 · output=2000 · total=2500 · cost_usd=$0.0125 ---
+  ```
+
+## Backward Compatibility
+
+All new fields are optional:
+- `status.json.metrics` may be absent for old runs.
+- `run_complete.metrics` may be absent for old runs.
+- `extension_error` and `model_change` events only present in jobs after 2026-04-08.
+- Enriched `compaction` and `retry` fields only present when those features are triggered.
+- Consumers must treat missing metrics as unknown, not zero.
+
+## Open Review Workflow
+
+For every new protocol-derived metric:
+1. Open RFC issue with sample RPC payloads and backward-compat notes.
+2. Update this matrix with source path and confidence/caveats.
+3. Add fixture-driven contract tests from recorded RPC traces.
+4. Require two approvals (maintainer + external reviewer) before stable surfacing.
+5. Keep added fields optional for at least one minor release window.
+
+## Implementation Notes (v2.0)
+
+### Timeline Event Schema
+
+See `src/specialist/timeline-events.ts` for canonical event type definitions:
+- `TimelineEventExtensionError` — extension error events
+- `TimelineEventModelChange` — model change events
+- `TimelineEventCompaction` — enriched with `tokens_before`, `summary`, `first_kept_entry_id`
+- `TimelineEventRetry` — enriched with `attempt`, `max_attempts`, `delay_ms`, `error_message`
+
+### Token Usage Display
+
+Token usage formatting uses shared helpers from `src/cli/format-helpers.js`:
+- `formatTokenUsageSummary()` — produces `input=X · output=Y · total=Z`
+- `formatCostUsd()` — produces `$0.0125` format
+
+Used by:
+- `src/cli/ps.ts` — list and inspect views
+- `src/cli/result.ts` — human-mode result display
+
+### Infrastructure Fixes
+
+**initSchema race condition (2026-04-08):**
+- Fixed DROP/RENAME race in `src/specialist/observability-sqlite.ts`
+- Gates specialist_jobs rebuild to only run when legacy columns are missing
+- Prevents "no such table: specialist_jobs" errors during concurrent init/read
+
+**Loader precedence (2026-04-08):**
+- Fixed in `src/specialist/loader.ts`
+- `config/specialists/` now takes precedence over `.specialists/default/`
+- Ensures `sp edit` changes are not overridden by stale default copies