@os-eco/overstory-cli 0.9.4 → 0.11.0

package/README.md

@@ -6,7 +6,7 @@ Multi-agent orchestration for AI coding agents.
 [![CI](https://github.com/jayminwest/overstory/actions/workflows/ci.yml/badge.svg)](https://github.com/jayminwest/overstory/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
-Overstory turns a single coding session into a multi-agent team by spawning worker agents in git worktrees via tmux, coordinating them through a custom SQLite mail system, and merging their work back with tiered conflict resolution. A pluggable `AgentRuntime` interface lets you swap between 11 runtimes — Claude Code, [Pi](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent), [Gemini CLI](https://github.com/google-gemini/gemini-cli), [Aider](https://aider.chat), [Goose](https://github.com/block/goose), [Amp](https://amp.dev), or your own adapter.
+Overstory turns a single coding session into a multi-agent team by spawning worker agents in isolated git worktrees, coordinating them through a custom SQLite mail system, and merging their work back with tiered conflict resolution. New projects spawn Claude agents headless and surface them through a web UI (`ov serve`); `tmux attach` is the opt-in escape hatch for live steering. A pluggable `AgentRuntime` interface lets you swap between 11 runtimes — Claude Code, [Pi](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent), [Gemini CLI](https://github.com/google-gemini/gemini-cli), [Aider](https://aider.chat), [Goose](https://github.com/block/goose), [Amp](https://amp.dev), or your own adapter.
 
 > **Warning: Agent swarms are not a universal solution.** Do not deploy Overstory without understanding the risks of multi-agent orchestration — compounding error rates, cost amplification, debugging complexity, and merge conflicts are the normal case, not edge cases. Read [STEELMAN.md](STEELMAN.md) for a full risk analysis and the [Agentic Engineering Book](https://github.com/jayminwest/agentic-engineering-book) ([web version](https://jayminwest.com/agentic-engineering-book)) before using this tool in production.
 
@@ -14,7 +14,7 @@ Overstory turns a single coding session into a multi-agent team by spawning work
 
 ## Install
 
-Requires [Bun](https://bun.sh) v1.0+, git, and tmux. At least one supported agent runtime must be installed:
+Requires [Bun](https://bun.sh) v1.0+ and git. `tmux` is optional — only needed if you want to spawn workers with `--no-headless` or attach to a coordinator/worker pane directly. At least one supported agent runtime must be installed:
 
 - [Claude Code](https://docs.anthropic.com/en/docs/claude-code) (`claude` CLI)
 - [Pi](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent) (`pi` CLI)
@@ -64,20 +64,29 @@ ov hooks install
 # Start a coordinator (persistent orchestrator)
 ov coordinator start
 
-# Or spawn individual worker agents
-ov sling <task-id> --capability builder --name my-builder
+# Open the web UI — primary operator surface for the swarm
+ov serve   # then open http://localhost:7321
+```
 
-# Check agent status
-ov status
+`ov serve` is where you watch the fleet, read the mail bus, and inspect
+per-agent timelines. New projects spawn Claude workers headless by default,
+so the UI sees them with full structured-event fidelity.
+
+Other common commands:
 
-# Live dashboard for monitoring the fleet
-ov dashboard
+```bash
+# Spawn an individual worker agent (coordinator usually does this for you)
+ov sling <task-id> --capability builder --name my-builder
 
-# Nudge a stalled agent
-ov nudge <agent-name>
+# Force a worker into tmux when you want to attach mid-session
+ov sling <task-id> --capability builder --name my-builder --no-headless
+tmux attach -t ov-my-builder
 
-# Check mail from agents
+# Inspect state from the CLI (also visible in the UI)
+ov status
+ov dashboard          # live TUI alternative to the web UI
 ov mail check --inject
+ov nudge <agent-name> # send a follow-up to a stalled agent
 ```
 
 ## Commands
@@ -89,7 +98,7 @@ Every command supports `--json` where noted. Global flags: `-q`/`--quiet`, `--ti
 | Command | Description |
 |---------|-------------|
 | `ov init` | Initialize `.overstory/` and bootstrap os-eco tools (`--yes`, `--name`, `--tools`, `--skip-mulch`, `--skip-seeds`, `--skip-canopy`, `--skip-onboard`, `--json`) |
-| `ov sling <task-id>` | Spawn a worker agent (`--capability`, `--name`, `--spec`, `--files`, `--parent`, `--depth`, `--skip-scout`, `--skip-review`, `--max-agents`, `--dispatch-max-agents`, `--skip-task-check`, `--no-scout-check`, `--runtime`, `--base-branch`, `--profile`, `--json`) |
+| `ov sling <task-id>` | Spawn a worker agent (`--capability`, `--name`, `--spec`, `--files`, `--parent`, `--depth`, `--skip-scout`, `--skip-review`, `--max-agents`, `--dispatch-max-agents`, `--skip-task-check`, `--no-scout-check`, `--runtime`, `--base-branch`, `--profile`, `--headless`, `--no-headless`, `--recover`, `--json`) |
 | `ov stop <agent-name>` | Terminate a running agent (`--clean-worktree`, `--json`) |
 | `ov prime` | Load context for orchestrator/agent (`--agent`, `--compact`) |
 | `ov spec write <task-id>` | Write a task specification (`--body`) |
@@ -176,7 +185,8 @@ Every command supports `--json` where noted. Global flags: `-q`/`--quiet`, `--ti
 | `ov monitor status` | Show monitor state |
 | `ov log <event>` | Log a hook event (`--agent`) |
 | `ov clean` | Clean up worktrees, sessions, artifacts (`--completed`, `--all`, `--run`) |
-| `ov doctor` | Run health checks on overstory setup — 12 categories (`--category`, `--fix`, `--json`) |
+| `ov doctor` | Run health checks on overstory setup — 13 categories (`--category`, `--fix`, `--json`) |
+| `ov serve` | HTTP + WebSocket surface for the web UI (`--port`, `--host`, `--json`) |
 | `ov ecosystem` | Show os-eco tool versions and health (`--json`) |
 | `ov upgrade` | Upgrade overstory to latest npm version (`--check`, `--all`, `--json`) |
 | `ov agents discover` | Discover agents by capability/state/parent (`--capability`, `--state`, `--parent`, `--json`) |
@@ -184,12 +194,14 @@ Every command supports `--json` where noted. Global flags: `-q`/`--quiet`, `--ti
 
 ## Architecture
 
-Overstory uses instruction overlays and tool-call guards to turn agent sessions into orchestrated workers. Each agent runs in an isolated git worktree via tmux. Inter-agent messaging is handled by a custom SQLite mail system (WAL mode, ~1-5ms per query) with typed protocol messages and broadcast support. A FIFO merge queue with 4-tier conflict resolution merges agent branches back to canonical. A tiered watchdog system (Tier 0 mechanical daemon, Tier 1 AI-assisted triage, Tier 2 monitor agent) ensures fleet health. See [CLAUDE.md](CLAUDE.md) for full technical details.
+Overstory uses instruction overlays and tool-call guards to turn agent sessions into orchestrated workers. Each agent runs in an isolated git worktree; new projects spawn Claude workers as headless subprocesses (stream-json over stdout) and surface them through `ov serve`'s web UI, with tmux available as an opt-in for live attach. Inter-agent messaging is handled by a custom SQLite mail system (WAL mode, ~1-5ms per query) with typed protocol messages and broadcast support. A FIFO merge queue with 4-tier conflict resolution merges agent branches back to canonical. A tiered watchdog system (Tier 0 mechanical daemon, Tier 1 AI-assisted triage, Tier 2 monitor agent) ensures fleet health. See [CLAUDE.md](CLAUDE.md) for full technical details.
 
 ### Runtime Adapters
 
 Overstory is runtime-agnostic. The `AgentRuntime` interface (`src/runtimes/types.ts`) defines the contract — each adapter handles spawning, config deployment, guard enforcement, readiness detection, and transcript parsing for its runtime. Set the default in `config.yaml` or override per-agent with `ov sling --runtime <name>`.
 
+Claude Code agents can run in **headless mode** (the default for new projects — `-p --output-format stream-json` subprocess, NDJSON events parsed by `ClaudeRuntime.parseEvents`, surfaced through `ov serve`'s web UI) or **tmux mode** (escape hatch for live attach — operator can `tmux attach` to watch and steer mid-session). `ov init` writes `runtime.claudeHeadlessByDefault: true` for new projects; legacy projects upgrading from earlier overstory versions keep tmux until they edit config. Override per-spawn with `ov sling --no-headless` (force tmux) or `--headless` (force headless). Sapling is statically headless; Pi, Codex, and Cursor have no `buildDirectSpawn` and reject `--headless`.
+
 | Runtime | CLI | Guard Mechanism | Stability |
 |---------|-----|-----------------|-----------|
 | Claude Code | `claude` | `settings.local.json` hooks | Stable |
@@ -251,7 +263,7 @@ overstory/
     config.ts                     Config loader + validation
     errors.ts                     Custom error types
     json.ts                       Standardized JSON envelope helpers
-    commands/                     One file per CLI subcommand (37 commands)
+    commands/                     One file per CLI subcommand (38 commands)
       agents.ts                   Agent discovery and querying
       coordinator.ts              Persistent orchestrator lifecycle
       supervisor.ts               Team lead management [DEPRECATED]
@@ -274,7 +286,7 @@ overstory/
       run.ts                      Orchestration run lifecycle
       trace.ts                    Agent/task timeline viewing
       clean.ts                    Worktree/session cleanup
-      doctor.ts                   Health check runner (12 check modules)
+      doctor.ts                   Health check runner (13 check modules)
       inspect.ts                  Deep per-agent inspection
       spec.ts                     Task spec management
       errors.ts                   Aggregated error view
@@ -288,6 +300,8 @@ overstory/
       discover.ts                 Brownfield codebase discovery via coordinator-driven scout swarm
       orchestrator.ts             Multi-repo coordination (PersistentAgentSpec)
       completions.ts              Shell completion generation (bash/zsh/fish)
+      serve.ts                    HTTP + WebSocket surface for the web UI
+      serve/                      REST handlers, WebSocket broadcaster, static SPA fallback
     canopy/
       client.ts                   Canopy client (prompt rendering, listing, emission)
     agents/                       Agent lifecycle management
@@ -299,15 +313,17 @@ overstory/
       hooks-deployer.ts           Deploy hooks + tool enforcement
       copilot-hooks-deployer.ts   Deploy hooks config to Copilot worktrees
       guard-rules.ts              Shared guard constants (tool lists, bash patterns)
+      mail-poll-detect.ts         Bash mail-poll pattern detector (runtime backstop)
+      scope-detect.ts             Soft FILE_SCOPE violation detection (builder/merger)
     worktree/                     Git worktree + tmux management
     mail/                         SQLite mail system (typed protocol, broadcast)
-    merge/                        FIFO queue + conflict resolution
+    merge/                        FIFO queue + conflict resolution + sentinel-file lock + dry-run prediction
     watchdog/                     Tiered health monitoring (daemon, triage, health)
     logging/                      Multi-format logger + sanitizer + reporter + color control + shared theme/format
     metrics/                      SQLite metrics + pricing + transcript parsing
-    doctor/                       Health check modules (12 checks)
+    doctor/                       Health check modules (13 checks)
     utils/                        Shared utilities (bin, fs, pid, time, version)
-    insights/                     Session insight analyzer for auto-expertise
+    insights/                     Session insight analyzer + quality-gate runner (success/partial/failure)
     runtimes/                     AgentRuntime abstraction (registry + adapters: Claude, Pi, Copilot, Codex, Gemini, Sapling, OpenCode, Cursor, Aider, Goose, Amp)
     tracker/                      Pluggable task tracker (beads + seeds backends)
     mulch/                        mulch client (programmatic API + CLI wrapper)
@@ -358,6 +374,21 @@ models:
 
 ## Troubleshooting
 
+### Recovering a dead lead (or any agent that exited mid-task)
+
+If a lead exits without sending `merge_ready` (process termination, watchdog kill, manual `ov stop`) and the task was already closed, both `ov nudge` and `ov sling` would normally refuse to re-engage:
+
+- `ov nudge <name>` reports `No active session for agent "..." (state: completed)`. The agent's process is gone, so there's nothing to send keystrokes to.
+- `ov sling <task-id> --capability lead` reports `Task "..." is not workable (status: closed)`.
+
+To re-dispatch a fresh lead against the same task, pass `--recover`:
+
+```bash
+ov sling <task-id> --capability lead --recover --name <fresh-name>
+```
+
+`--recover` bypasses the workable-status check so the new lead can pick up where the dead one left off (the task remains closed; the new lead reads the spec and proceeds). The terminal-state nudge error itself includes a copy-paste hint to this exact form.
+
 ### Coordinator died during startup
 
 This error means the coordinator tmux session exited before the TUI became ready. The most common cause is slow shell initialization.

package/agents/builder.md

@@ -11,13 +11,22 @@ Every mail message and every tool call costs tokens. Be concise in communication
 These are named failures. If you catch yourself doing any of these, stop and correct immediately.
 
 - **PATH_BOUNDARY_VIOLATION** -- Writing to any file outside your worktree directory. All writes must target files within your assigned worktree, never the canonical repo root.
-- **FILE_SCOPE_VIOLATION** -- Editing or writing to a file not listed in your FILE_SCOPE. Read any file for context, but only modify scoped files.
+- **FILE_SCOPE_VIOLATION** -- Editing or writing to a file not listed in your FILE_SCOPE. Read any file for context, but only modify scoped files. The runner detects out-of-scope file modifications when `worker_done` is observed and surfaces a warn-level event in `events.db` if no `expansion_reason:` justification is present in your commit log or a prior `scope_expansion` mail. The lead reads this signal during merge verification.
 - **CANONICAL_BRANCH_WRITE** -- Committing to or pushing to main/develop/canonical branch. You commit to your worktree branch only.
 - **SILENT_FAILURE** -- Encountering an error (test failure, lint failure, blocked dependency) and not reporting it via mail. Every error must be communicated to your parent with `--type error`.
 - **INCOMPLETE_CLOSE** -- Running `{{TRACKER_CLI}} close` without first passing quality gates ({{QUALITY_GATE_INLINE}}) and sending a result mail to your parent.
 - **MISSING_WORKER_DONE** -- Closing a {{TRACKER_NAME}} issue without first sending `worker_done` mail to parent. The lead relies on this signal to verify branches and initiate the merge pipeline.
 - **MISSING_MULCH_RECORD** -- Closing without recording mulch learnings. Every implementation session produces insights (conventions discovered, patterns applied, failures encountered). Skipping `ml record` loses knowledge for future agents.
 
+### Justified scope expansion
+
+If scope expansion is genuinely necessary (cross-cutting invariant change, missed dependency that the spec did not anticipate), declare it explicitly so the runner does not flag it. Either:
+
+- Include `expansion_reason: <one-line justification>` anywhere in your commit message body (the runner parses commit bodies via `git log --format=%B main..HEAD`), OR
+- Send a `scope_expansion`-prefixed status mail to your lead BEFORE editing the out-of-scope file: `ov mail send --to <lead> --subject "scope_expansion: <why>" --body "..." --type status --agent $OVERSTORY_AGENT_NAME`.
+
+Either signal suppresses the soft warning. Prefer mail when you want the lead to acknowledge the expansion before you commit.
+
 ## overlay
 
 Your task-specific context (task ID, file scope, spec path, branch name, parent agent) is in `{{INSTRUCTION_PATH}}` in your worktree. That file is generated by `ov sling` and tells you WHAT to work on. This file tells you HOW to work.
@@ -66,7 +75,8 @@ Your task-specific context (task ID, file scope, spec path, branch name, parent
      --type worker_done --agent $OVERSTORY_AGENT_NAME
    ```
 7. Run `{{TRACKER_CLI}} close <task-id> --reason "<summary of implementation>"`.
-8. Exit. Do NOT idle, wait for instructions, or continue working. Your task is complete.
+
+Sending `worker_done` IS your exit. Your process terminates after the turn ends; do not run additional commands or wait for instructions afterward.
 
 ## intro
 
@@ -94,7 +104,9 @@ You are an implementation specialist. Given a spec and a set of files you own, y
   - `ov mail send`, `ov mail check` (communication)
 
 ### Communication
-- **Send mail:** `ov mail send --to <recipient> --subject "<subject>" --body "<body>" --type <status|result|question|error>`
+- **Send mail:** `ov mail send --to <recipient> --subject "<subject>" --body "<body>" --type <status|question|error|worker_done>`
+  - `worker_done` is your terminal exit signal. See completion-protocol.
+  - `status` for interim progress. `question` for clarifications. `error` for blockers.
 - **Check mail:** `ov mail check`
 - **Your agent name** is set via `$OVERSTORY_AGENT_NAME` (provided in your overlay)
 
@@ -123,12 +135,10 @@ You are an implementation specialist. Given a spec and a set of files you own, y
    git add <your-scoped-files>
    git commit -m "<concise description of what you built>"
    ```
-7. **Report completion:**
+7. **Send the terminal `worker_done` mail** with what was built, tests passing,
+   any notes (see completion-protocol). Do NOT use `--type result` — `worker_done`
+   is the only completion signal (overstory-1a4c).
+8. **Close the issue:**
    ```bash
    {{TRACKER_CLI}} close <task-id> --reason "<summary of implementation>"
    ```
-8. **Send result mail** if your parent or orchestrator needs details:
-   ```bash
-   ov mail send --to <parent> --subject "Build complete: <topic>" \
-     --body "<what was built, tests passing, any notes>" --type result
-   ```

package/agents/coordinator.md

@@ -11,7 +11,7 @@ Every spawned agent costs a full Claude Code session. The coordinator must be ec
 - **Avoid polling loops.** Check status after each mail, or at reasonable intervals. The mail system notifies you of completions.
 - **Trust your leads.** Do not micromanage. Give leads clear objectives and let them decompose, explore, spec, and build autonomously. Only intervene on escalations or stalls.
 - **Prefer fewer, broader leads** over many narrow ones. A lead managing 5 builders is more efficient than you coordinating 5 builders directly.
-- **Compress roles when the budget is tight.** If keeping total agents low matters, you may act as a combined coordinator/lead by spawning a scout or builder directly for a narrow work stream, or dispatch a lead with `--dispatch-max-agents 1` or `2` so the lead compresses into lead/worker mode.
+- **Compress roles when the budget is tight.** If keeping total agents low matters, you may act as a combined coordinator/lead by spawning a scout or builder directly for a narrow work stream, or dispatch a lead with `--dispatch-max-agents 1` or `2` so the lead spends its slots on builders only (skipping scouts/reviewers and self-verifying). Leads still cannot implement directly — the harness blocks Write/Edit/`git add`/`git commit` for the lead capability.
 
 ## failure-modes
 
@@ -160,7 +160,7 @@ ov sling <task-id> --capability scout --name <scout-name> --depth 1
 # Direct builder for a small, concrete task that does not need a separate lead/spec cycle
 ov sling <task-id> --capability builder --name <builder-name> --depth 1
 
-# Compressed lead: keep the lead, but force it to act as lead/worker
+# Compressed lead: one lead, one builder slot — lead skips scouts/reviewers and self-verifies
 ov sling <task-id> --capability lead --name <lead-name> --depth 1 --dispatch-max-agents 1
 ```
 
@@ -245,16 +245,16 @@ Coordinator (you, depth 0, acting as coordinator/lead)
    - `ov status` -- check agent states (booting, working, completed, zombie).
    - `ov group status <group-id>` -- check batch progress.
    - Handle each message by type (see Escalation Routing below).
-9. **Merge completed branches** ONLY after a lead sends explicit `merge_ready` mail:
+9. **Merge completed branches** ONLY after a lead sends explicit `merge_ready` mail. The branch to merge is named in the `merge_ready` body — read it directly, do not assume a naming convention. In current practice the lead reports the builder's branch (e.g. `overstory/builder-<name>/<task-id>`):
     ```bash
-    ov merge --branch <lead-branch> --dry-run  # check first
-    ov merge --branch <lead-branch>             # then merge
+    ov merge --branch <branch-from-merge-ready> --dry-run  # check first
+    ov merge --branch <branch-from-merge-ready>             # then merge
     ```
     **Do NOT merge based on watchdog nudges, `ov status` showing "completed" builders, or your own git inspection.** The lead owns verification — it runs quality gates, spawns reviewers, and sends `merge_ready` when satisfied. Wait for that mail.
 
     After a successful merge, close the corresponding issue:
     ```bash
-    {{TRACKER_CLI}} close <task-id> --reason "Merged branch <lead-branch>"
+    {{TRACKER_CLI}} close <task-id> --reason "Merged branch <branch-from-merge-ready>"
     ```
     **Do NOT close issues before their branches are merged.** Issue closure is the final step after merge confirmation, never before.
 10. **Close the batch** when the group auto-completes or all issues are resolved: