@staix/agent-hub 0.12.20 → 0.12.22

package/docs/operations.md

@@ -1,6 +1,6 @@
 # Operations guide
 
-This guide describes ahub 0.12.20 and control protocol 16. Live verification
+This guide describes ahub 0.12.22 and control protocol 16. Live verification
 results and remaining prerequisites are recorded separately in [the smoke ledger](smoke.md).
 
 ## Command help
@@ -49,15 +49,17 @@ counts `needs_review` deliveries, in the attention and failure tones.
 
 Peers, Approvals, Tasks and Queue are tables with a header row. A column is as
 wide as its widest cell, up to a third of the terminal; the last column takes
-the rest and is cut with `...`.
+the rest and is cut with `...`. In Tasks, TITLE takes the rest and the STAGE
+meter keeps a fixed column at the right edge.
 
 | Panel | Columns |
 |---|---|
-| Peers | PEER STATE LINK Q ! REVIEW PAUSE QUOTA MODEL |
+| Peers | PEER STATE MODE LINK Q ! REVIEW PAUSE QUOTA MODEL |
 | Approvals | ID PEER LEFT TITLE |
-| Tasks | ID STATE OWNER REVIEWER CLASS AGE TITLE |
+| Tasks | ID STATE OWNER REVIEWER CLASS AGE TITLE STAGE |
 | Queue | ID PEER STATE REV AGE |
 
+The MODE column appears only when at least one peer has a non-`ask` mode.
 Zero counters and unknown values read `-`. Durations read `45s`, `12m`, `3h05m`
 or `2d03h`; quota resets and budget pauses read `in 2h13m`, and a pause by the
 console user reads `user`. When a peer's status carries #205's optional
@@ -83,14 +85,43 @@ showed it. The footer's last line lists only the keys that act in the current
 mode, panel and state. A notice shows on the footer's second line until the next
 key or for about ten seconds: errors and refusals (a refused command, a request
 error, the size refusal) in the failure tone, notices that only inform (a closed
-or missing selection) in the attention tone. With a detail open, `p`, `r` and `a`
+or missing selection) in the attention tone. With a detail open, `p`, `r`, `m` and `a`
 act on the item the detail shows. `:` enters a command; its output lines
-start at column 4, as message bodies do. Assignment, delivery resolution and
-allow decisions require confirmation; delivery resolution requires a reason. Tasks use the same public redaction as the board. Panels need at least
-80 columns by 24 rows; smaller terminals stay in stream mode. Task and queue
-polling runs only while the corresponding panel is visible. Leaving restores
+start at column 4, as message bodies do. Assignment, delivery resolution, allow
+decisions and a never-ask permission mode require confirmation; delivery resolution requires a reason. Tasks use the same public redaction as the board. Panels need at least
+80 columns by 24 rows; smaller terminals stay in stream mode. Full task and queue reads run only in panels; stream progress uses the existing
+status state counts and never fetches the board. Leaving restores
 the terminal and returning from panels replays the bounded stream buffer.
 
+The Tasks panel summarizes the whole public board with an approved/all fraction,
+a 20-cell ASCII bar and state counts. Proposed tasks with unapproved or missing
+`after` dependencies count as waiting; PII stubs count normally. Each task has a
+four-cell stage meter (proposed, in progress, review, approved). Changes requested
+returns to in progress and uses `!` plus the failure tone. Narrow panels retain
+the fraction/bar before optional counts, which appear in fixed importance order:
+changes requested, review, waiting, in progress, proposed, then unknown when nonzero. Narrow widths drop
+only the suffix. Bar cells and percentages round down so open tasks never imply
+completion. Stream `tasks N/M approved` lives on the approvals footer line, leaving the
+peer/queue footer line intact. Operator notices or an already-full approvals line
+take precedence over this optional count. A brief notice hides it only until the
+notice's ten-second TTL expires, then the retained known count returns on redraw; the scroll
+region starts at row 1 to preserve terminal scrollback. All existing status state
+counts contribute to its total, including an unfamiliar state. Until a successful
+status read it says loading; later failures preserve the prior count with a notice.
+Panels retain their existing full-board refresh and loading/error handling.
+
+The dashboard Task board shows the same shared public-board progress model,
+including state counts, waiting, stacked segments and a labeled four-stage track
+per row. Filtering rows does not filter overall progress. The existing signature guard
+updates the live region only when progress changes. The stacked SVG fills the
+section width; its text legend carries every state count. Theme offers System,
+Light and Dark; it is applied before first paint and stored in a host-scoped loopback cookie
+(`Path=/`, `SameSite=Strict`, one-year lifetime), shared across dashboard ports.
+Blocked cookie access defaults to System on reload and never prevents operation. System follows
+OS color preference through CSS. Both inline scripts and the style retain CSP
+hashes; there are no external chart/theme assets. Live theme/readability inspection
+is tracked in `docs/smoke.md` and remains unverified until performed.
+
 Console color policy is `--color=auto|always|never`, with `auto` as the default.
 Auto enables color only when both input and output are TTYs, `TERM` is not
 `dumb`, and `NO_COLOR` is empty or absent. Explicit `always` overrides these
@@ -105,11 +136,30 @@ management or the final reset. Invalid values fail before connecting.
 | Waiting and attention | Yellow | Busy/paused peer, pending approval, confirmation, review/ready task, important priority |
 | Failure and intervention | Red | Failed/check-failed task, undeliverable/overflow, `needs_review` queue, denial request or expired/cancelled approval |
 | Metadata | Bright black | Ages, remaining approval time and rules |
+| Stream peer names | Claude bright blue (94), Codex bright cyan (96); all others default | Fixed map, no hashing or themes |
+| Stream `task` keyword | Magenta | Task event keyword |
+| Stream numbers | Bold default foreground (1) | Percentages, durations and measured times |
+| Stream issue references | Underlined default foreground (4) | `#232` |
+| Stream task references | Magenta underlined (4;35) | `#1` following `task`/`Task`, `[task #1]`, `[review #1]` |
 
 Offline peers, primary titles, action details and message body lines keep the
 terminal's default foreground. Labels and prompts remain readable without
-color. Stream styling applies only to the header, using structured event data;
-message text cannot choose a color. A local denial is shown as requested, not
+color. Stream styling applies only to meaningful tokens in the first line: peer names,
+`task`, numbers and references. Ordinary words keep the default foreground.
+Task prefixes and bracketed `[review #N]` identify task references
+case-insensitively; prose `review #N` and other `#N` tokens are issue references. Legacy assign/which notices lack typed reference
+ranges and use the issue tone until structured spans identify them. The dated
+#239 palette reserves green/yellow/red for states, cyan for navigation and
+blue/cyan for Claude/Codex. Tokens use plain magenta, underlining and bold default
+foreground; no token SGR equals a state or peer SGR. The `task` keyword is never
+bold. Only hub-written peer slots are colored, not title words or decline reasons.
+State words and structural markers retain their event tone. Numeric-looking
+identifiers stay plain. Wrapped headers normalize tabs before tokenization; plain headers preserve
+their original tabs with color enabled or disabled.
+A projection miss on the first physical header keeps only structural tones.
+A missed continuation and every following continuation are fully plain; their
+text cannot supply structural tones. Projection carries its span index forward
+instead of rescanning earlier spans for every line. Bodies and command output stay plain. A local denial is shown as requested, not
 as a confirmed receipt; a remote answered closure has no option-kind metadata
 and is not guessed to be a denial. A fixed palette is applied after terminal
 control sanitization. Width, clipping, wrapping and cursor placement use plain
@@ -129,8 +179,8 @@ recorded separately in the smoke ledger; fake-terminal tests establish policy,
 geometry, sanitization and restoration only.
 
 The command input accepts existing status, board, task, review, say, pause,
-resume, budget, queue, permit, ask, remember, route, turns, undo, check-path and
-report operations. It executes an argument vector with closed stdin. Lifecycle,
+resume, budget, queue, permit, permission, ask, remember, route, turns, undo,
+check-path and report operations. It executes an argument vector with closed stdin. Lifecycle,
 launch, nested console, setup, UI, logs and tail commands are refused.
 
 ## Conducting a team from Claude Code or Codex
@@ -163,7 +213,8 @@ final response in the native TUI. Broadcasting implementation instructions can
 cause an otherwise unassigned owner to claim duplicate work.
 
 The person answers approvals in the console, resolves `needs_review` deliveries
-with `ahub queue resolve`, and handles budget overrides and hub lifecycle.
+with `ahub queue resolve`, stops a headless peer with `ahub stop`, and handles
+budget overrides and hub lifecycle.
 Running these commands from an agent shell is refused; the CLI retains the
 agent's identity even when invoked through a shell tool.
 
@@ -177,9 +228,11 @@ off withdraws pending feed notices.
 
 For a Claude conductor, `ahub claude` also observes native session and turn
 boundaries when facts injection and task-idle sweeps are off. Keep the managed
-hooks enabled to measure completion and supervision usage. Passing your own
-`--settings` takes precedence and produces a warning when it replaces that
-observation. The launcher records its private session identity in ordinary
+hooks enabled to measure completion and supervision usage. Your own
+`--settings` (given once, a JSON object or file) keeps that observation: the
+launcher adds the hub's hooks to its hooks and warns only that the status line
+tee is off; one that sets `disableAllHooks` is refused. The launcher records its
+private session identity in ordinary
 terminals too; this does not grant terminal-recovery authority.
 Ordinary Claude sessions with turn-free facts or task-idle sweeps enabled get the
 same session/start observation. Other ordinary launches remain non-opt-in.
@@ -216,7 +269,8 @@ may request an approval.
 
 `.agenthub/config.json` can be committed and shared. The fields that choose
 what the hub runs, which files it sends as credentials, where task text goes,
-or how far the local worker's sandbox reaches (`kimi_cmd`, `codex_bin`,
+how far the local worker's sandbox reaches, or how often a peer asks before it
+acts (`kimi_cmd`, `permission_modes`, `codex_bin`,
 `pi.cmd`, `checks`, `mlx.bin`, `mlx.runtimeDir`, `mlx.modelPath`, `omniroute.urls`,
 `omniroute.access_hosts`, the `omniroute` key files, `memory.worker_url`,
 `local.read_allow`, `local.bash_network`, `local.network_allow`) are machine-local:
@@ -233,6 +287,19 @@ project and a temp dir of their own (`TMPDIR`, made for each command and
 removed when it ends; one left by a hub crash, named `ahub-cmd-*`, goes with
 the OS temp cleanup), and nothing else; the shared temp dirs are closed.
 
+Some names stay closed to those commands, and to Pi's, wherever they may
+otherwise write: anything named `.git` at any depth, the `config` and `hooks`
+directly under a `.git`, `commondir` below it, and `.agenthub` anywhere in the
+project. A submodule's git directory under `.git/modules/` is not covered yet
+(#281). A command can therefore not create, rename or remove a `.git`:
+`git init`, `git clone` and `git worktree add` do not run inside the sandbox,
+and neither do `git worktree remove`, `git worktree prune` of a stale record,
+`git clean -ffd` over a nested repository or `rm -rf` of a directory that
+holds a `.git`. `git add`, `commit`, `stash`, `checkout` and `gc` work, and
+`.gitignore` and `.github/` stay writable; a commit that your git configuration
+signs with a key under `~/.ssh` or `~/.gnupg` fails, because the sandbox cannot
+read those directories. Run a refused command yourself, in a terminal.
+
 Network is off unless `local.bash_network` says otherwise. With `true` it goes
 only through the hub's egress proxy on a loopback port (0.11): commands get
 `HTTPS_PROXY` and the other proxy variables, the proxy opens HTTPS (`CONNECT`,
@@ -399,7 +466,10 @@ offered to nobody, cannot be claimed, accepted or marked done, and `ahub route
 explain <id>` says what it waits for. When the last one is approved, the task
 goes through assignment like a new one; if the hub stopped before it got that
 far, the task is offered within a minute of a peer that can take it attaching
-to the next run.
+to the next run. An offer is used up only by an assignment, after an approval
+and on the timer alike. If no peer can take it or a ready/assignment write fails,
+the task stays eligible; the timer adds no history or notices while nobody can
+take it.
 `ahub board --ready` and
 `hub_task_list {ready: true}` list the proposed tasks with nothing left to wait
 for. Dependencies are fixed when a task is proposed and can only name tasks that
@@ -498,6 +568,37 @@ turn, otherwise the peer's only `in_progress` task. Everything else is reported
 unattributed, and records from before 0.12.18 as a separate bucket; neither is
 redistributed (rules: [events](events.md#per-task-usage-reports)).
 
+## Research records
+
+With `"research": { "enabled": true }` in `.agenthub/config.json` (off by default), the hub keeps one record per
+approved task in `~/.agenthub/research/`, shared by every project that opts in: outcome, review rounds, rework, failed
+checks, tokens and time per peer and model, never task or message text ([schema](research.md)).
+
+```bash
+ahub research --since 30d          # success, first-pass, rework and check-failure rates; tokens and wall time per task
+ahub research --all --json         # every opted-in project on this machine
+ahub research export --format csv  # the records, for your own analysis
+ahub research backfill             # build this project's records once from events.jsonl
+ahub task label 12 reverted        # a later verdict (ok, regressed, reverted, incomplete, wrong, abandoned)
+```
+
+## Benchmarks
+
+In a git repository kept for benchmarks (`"bench": { "enabled": true }` in its `.agenthub/config.json`, a clean tree),
+`ahub bench` runs a suite of tasks against the peers attached to its hub and compares configurations: pass rate, first
+pass, tokens and wall time per attempt, with bootstrap intervals between arms. Each attempt resets the tree to the
+task's commit, so never enable it in a project you work in. The project must be the root of its repository with
+nothing tracked under `.agenthub/`, and the suite file lives outside its tree ([suite format and store](bench.md)).
+
+```bash
+ahub bench run ../suites/suite.json --arm baseline --repeat 5   # a person's: resets the tree, proposes the tasks
+ahub bench status                                     # the run in progress
+ahub bench compare baseline ripwire-on                # arms side by side, inconclusive below 5 attempts each
+ahub bench export --format csv
+```
+
+The dashboard's Benchmarks section shows the latest runs and, per suite with two or more arms, the comparison.
+
 ## Turns and undo
 
 In a git work tree the hub snapshots the project's tracked and unignored files (the
@@ -607,8 +708,9 @@ stays the default until an evaluation says otherwise (`docs/cooperbench.md`).
   held.
 - Verified context path. A hub-launched Claude session (`ahub claude`) gets the
   hub's hook before and after every tool call and at the end of each turn, in
-  its `--settings` next to the status line tee (a `--settings` of your own turns
-  them off). Codex gets context by steer into its running turn. Until a fact or a
+  its `--settings` next to the status line tee (a `--settings` of your own keeps
+  them, added to its own hooks, and loses only the tee). Codex gets context by
+  steer into its running turn. Until a fact or a
   one-line probe has been read back, the peer counts as unverified: for Claude
   the row Claude Code writes in its transcript for the hook's additional
   context, matched by tool use id and the offer's id; for Codex the steered
@@ -753,6 +855,107 @@ stays the default until an evaluation says otherwise (`docs/cooperbench.md`).
   turns the stale-notice drop off, for a controlled comparison only (the #106
   ablation in `docs/cooperbench.md`); the hub logs it at start.
 
+## Permission modes for running peers
+
+Use a plain terminal to inspect or change a running peer:
+
+```bash
+ahub permission
+ahub permission local ask-when-needed
+ahub permission kimi never-ask --yes
+ahub permission kimi ask
+```
+
+In the console Peers panel, `m` opens the three modes. Never-ask requires `y`,
+even for a typed command carrying --yes. Its confirmation says whether the peer
+uses its own tools outside the hub sandbox (Kimi), the hub sandbox/path guard/
+denylist (Pi and local), or native vendor boundaries (Claude and Codex).
+
+| Peer | ask-when-needed | never-ask | ask |
+| --- | --- | --- | --- |
+| Kimi Code CLI | ACP yolo | ACP auto | ACP default |
+| Codex | on-request for every outgoing turn/start | never for every outgoing turn/start | Restore the known native policy once, then stop overriding |
+| Claude | Allow resolving project file tools outside .agenthub/.git/.claude/.codex/.qwen/.kimi/.pi/.mcp.json | Hook allow for every tool | No hook decision |
+| Pi and local | Grant write/edit outside native agent configuration paths; read never asks; shell/git writes ask | Grant each tool once within the existing sandbox/guard | Normal approval handling |
+
+Kimi's map applies only to the actual Kimi Code CLI identity, not another ACP
+agent configured in kimi_cmd. Qwen and other ACP agents start fresh/resumed
+unmanaged without mode changes or mode-based startup refusals; runtime changes
+are refused until their own map exists.
+A set_mode timeout leaves the peer offline with mode unknown; inspect and
+restart before using it. Missing ids and refused startup changes name the mode.
+The benign Kimi live check verifies file edits and harmless shells in both
+modes without requests; risky-action behavior remains unverified.
+
+Claude's hook is installed in every managed launch. File grants use canonical
+real paths under the authenticated project root. Missing targets, symlink
+escapes, metadata directories and uncertain wildcard targets leave the native
+rules in charge. Grep/Glob path-bearing filters cannot bypass the exclusions.
+Other tools retain native decisions, which may already allow a harmless shell.
+Permission-only PreToolUse never marks Claude busy; former observation hooks
+retain their native-turn role. Settings are passed as a 0600 state file retained
+until that native launch exits. Live/unknown previous launchers keep their files;
+verified dead wrapper and native identities permit crash cleanup. Managed direct
+binary launches record their native PID while running; crash cleanup also requires
+that PID to be absent from the process table. Scripts/interpreters and unavailable
+process evidence retain the settings file. Every launch atomically replaces
+`claude-launch.json` with its current managed hook/session identity. Native PID
+proof is separate in a private `claude-native-<binding-hash>.json` file, so an old
+callback cannot overwrite a newer launch record. Only that native proof uses
+`claude-launch.lock`, published with its signed owner already written. It waits
+up to 250 ms for a verified live owner. An empty, unsigned or unverifiable lock
+is retained and disables native identity/crash cleanup for this launch; the
+warning names the path and what is off. Managed hooks and session observations
+still use the newly published launch record. Inspect the named lock and verify
+its owner has ended before removing it; a signed dead lock is recovered. Without
+separate native identity proof, old settings remain for manual inspection. Settings carry
+no inline caller values in argv. Status reports an unverified Claude hook as
+unverified. Unattended native sessions cannot take a runtime mode; restart
+without that native flag. Codex must be behind the hub proxy, and returning to
+ask while attached requires a known native approval baseline. If unavailable,
+close the TUI, then run `ahub permission codex ask` before starting `ahub codex`.
+For a known baseline, a detached proxy keeps restoration owed per thread and sends
+that policy once on the resumed thread's next accepted turn. Visiting another thread
+does not clear the debt. The restoring turn uses that kept baseline even if the
+TUI echoes the resume response's sticky override, whether ask was selected before
+or after that resume. Rejected turns retain it. The protection covers that one
+restoring turn. A later TUI turn that repeats the policy looks exactly like a
+person's explicit native policy choice and is passed through; the proxy cannot
+resolve that ambiguity. An explicit choice on the restoring turn is delayed
+until the following turn. A baseline that was never reported
+is explained in the CLI reply, and that old thread keeps its last policy.
+An offline or absent peer accepts
+`ahub permission <peer> ask` to clear its hub choice. Codex --unattended launch
+is refused under a non-ask choice; clear it with `ahub permission codex ask`
+first. A pending never-ask default cannot be confirmed for an unattended launch.
+
+Project defaults use permission_modes in .agenthub/config.json or
+.agenthub/config.local.json. Keys merge per peer; a local ask overrides the
+same peer's earlier default, while an empty local block preserves it. Every
+non-ask default start logs its mode and contributing filename. Automatic
+config effects stop at ask-when-needed. Git-tracked defaults are ignored and never
+offered for confirmation. Never-ask from an applied untracked file starts in ask
+and appears as a pending source-labelled console question at hub start.
+Only its separate console y enables that default; there is no boot --yes
+bypass. Only explicit n declines and keeps ask for that hub. Esc defers without declining
+and leaves approval keys usable; a refused y is not reoffered until its state changes. A successful runtime change removes that pending default. Runtime choices expire when the hub
+stops, and a later restart asks again for a never-ask config default.
+
+The CLI marker and console-role checks are operating policy, not a hostile-agent
+boundary. An agent able to run unrestricted shells may clear its markers,
+use --yes, or read the control token and act as the console. Hub tools,
+conductor operations and ordinary agent messages have no direct mode-changing
+or startup-confirmation operation, but those checks do not contain such a
+shell. Review the boundaries in [security](security.md) before choosing a mode.
+Modes never alter native sandboxes, hub path guards or denylists.
+
+Permission mode starts reconcile the current operator choice after native startup before bus delivery. Each non-ask start logs the mode and its config or runtime source. Dropped git-tracked defaults never appear as startup confirmations. Non-Kimi ACP peers run unmanaged, preserving their native fresh/resumed mode; permission commands refuse changes until a vendor map exists.
+
+Status shows non-ask modes and pending defaults, and each runtime change emits
+permission_mode (peer/from/to) with no tool arguments under events schema 1.
+The optional control requests preserve PROTOCOL; an older hub answers unknown
+and the CLI advises upgrading.
+
 ## Approvals and pauses
 
 Inspect permission requests in the terminal:
@@ -771,9 +974,11 @@ ahub permit <request-id> allow
 
 Use the exact option shown by `ahub tail`; do not approve an unresolved or
 unexpected request. On macOS a waiting request also raises a desktop
-notification that names the peer and, for Kimi, the tool, never what it would
-run. An unanswered request is cancelled after `approvals.timeout_s` (default
+notification that names the peer and, for Kimi, Pi and the local worker, the tool,
+never what it would run. An unanswered request is cancelled after `approvals.timeout_s` (default
 120, 30 to 3600) in `.agenthub/config.json`, and the console and log say so.
+Pi and the local worker are told that nobody answered, and two such requests in
+a row end their turn ([Pi/local unanswered approvals](#pilocal-unanswered-approvals)).
 Set `approvals.notify` to `false` to turn notifications off; a hub started
 without a project config file raises none. While Kimi waits for an answer its
 turn is kept alive, so a timeout longer than the inactivity watchdog does not
@@ -832,6 +1037,9 @@ has only `ahub status`, `ahub tail` and the task board.
 
 Resolution requires the current observed revision. A retry closes the old
 record and creates one linked attempt. Stale or conflicting resolutions fail.
+A refused or failed resolution puts back only that peer's queue and preface
+and keeps every pause, including budget and conductor holds. A storage failure
+stops the bus; a validation refusal leaves it running under the existing holds.
 If dispatching or accepted work has an uncertain outcome after a crash,
 the system marks it `needs_review`. Establish evidence before choosing
 `completed`, `retry`, or `discard`. Timeouts,
@@ -876,11 +1084,15 @@ The daemon waits for shutdown work that must survive termination. Afterward,
 run `ahub status` or `ahub up` and read the project state before restarting
 peers. A stopped daemon does not delete task records or the durable journal.
 Do not kill a native terminal by PID or start a replacement while ownership is
-uncertain.
+uncertain. To stop one headless peer and keep the hub running, use
+`ahub stop <peer>` ([Stop one headless peer](#stop-one-headless-peer)).
 
 Shutdown is bounded: once it begins, a daemon that cannot finish within
 15 seconds exits anyway, and a peer that refuses to stop is logged rather than
-allowed to block state cleanup. A native Pi TUI owner that does not exit
+allowed to block state cleanup. A managed Pi tool that does not settle within
+the stop grace does not block the teardown either: the stop goes on
+(shutdown, owner teardown, process stop) and reports the failure at the end.
+A native Pi TUI owner that does not exit
 gracefully is terminated by its verified process identity (never a bare PID),
 so a restarted hub never launches beside a survivor. A hub whose project root
 or state directory was deleted stops itself within about 10 seconds; the
@@ -910,8 +1122,10 @@ The runtime reset (the default):
 - drops the agent session resume pointers `sessions.json`,
   `claude-session.json` and `claude-context.json`, so the next launch starts
   new agent sessions. Pi transcripts under `pi-sessions/` stay;
-- removes the manifest (`status.json`, `control-token`, `hub.pid`) a hub left
-  when it did not stop cleanly, once no process behind it is alive;
+- removes the manifest (`status.json`, `control-token`, and `hub.pid` from a
+  hub of 0.12.20 or older) a hub left when it did not stop cleanly, once no
+  process behind it is alive (a pid that now belongs to another process counts
+  as gone when the manifest carries the hub's signature, from 0.12.21);
 - keeps the board (tasks, reviews, outcomes, turns, touches), `hub.log`,
   `events.jsonl` and `cli-audit/`, recovery records, execution budgets,
   configuration and files the hub does not own.
@@ -999,23 +1213,38 @@ Rows without a live process are stale registrations; forget them with
 
 ## Upgrade and crash recovery
 
+On a terminal, one command reviews, applies and follows an upgrade:
+
+```bash
+ahub upgrade
+```
+
+Without `--to` the target is the registry's latest release. When that release is
+newer than the installed CLI, the CLI names the release's own coordinator
+(`bun x --package @staix/agent-hub@<version> ahub upgrade --to <version>`), asks,
+and hands over to it, so the version is never typed twice and an older CLI never
+coordinates. From an installed CLI older than 0.12.22, start it once by hand:
+`bunx --package @staix/agent-hub@<version> ahub upgrade`. The screens are
+described under [Interactive upgrade](#interactive-upgrade); the rest of this
+section is what they run.
+
 Upgrade running projects with the target release's own coordinator. It accepts
 a running source on control protocol 9 (0.6.x), 10 (0.7.0 through 0.12.0),
-11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20), and only
+11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 through 0.12.22), and only
 a target on its own protocol, so the target's coordinator fits every supported
 source and carries every recovery fix released up to it. Protocol 8 and older
 (0.5.x and earlier) are refused as `manual-bootstrap-required`. Run from the
 project directory, without replacing the global CLI first:
 
 ```bash
-bunx --package @staix/agent-hub@0.12.20 ahub upgrade --to 0.12.20 --dry-run
-bunx --package @staix/agent-hub@0.12.20 ahub upgrade --to 0.12.20 --yes
+bunx --package @staix/agent-hub@0.12.22 ahub upgrade --to 0.12.22 --dry-run
+bunx --package @staix/agent-hub@0.12.22 ahub upgrade --to 0.12.22 --yes
 ```
 
 | Running now | Coordinator to use |
 | --- | --- |
 | 0.6.x (protocol 9) | the target's, through `bunx` as above |
-| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 (protocol 16) | the target's, through `bunx` as above |
+| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 through 0.12.22 (protocol 16) | the target's, through `bunx` as above |
 | any supported source, with the installed CLI already at the target | `ahub upgrade` below, which is the same coordinator |
 | 0.5.x or earlier (protocol 8 and older) | not supported: bootstrap by hand with the matching CLI |
 
@@ -1047,21 +1276,27 @@ projects first:
 
 ```bash
 ahub restart --dry-run
-ahub upgrade --to 0.12.20 --dry-run
+ahub upgrade --to 0.12.22 --dry-run
 ```
 
 Apply only after reviewing the plan:
 
 ```bash
 ahub restart --yes
-ahub upgrade --to 0.12.20 --yes
-ahub recovery status <operation-id>
-ahub recovery resume <operation-id>
-ahub recovery abort <operation-id>
-ahub recovery dispose <operation-id> --fresh-session <peer> --reason <text>
-ahub recovery dispose <operation-id> --stop-and-archive --reason <text>
+ahub upgrade --to 0.12.22 --yes
+ahub recovery status [<operation-id>]
+ahub recovery resume [<operation-id>]
+ahub recovery abort [<operation-id>]
+ahub recovery dispose [<operation-id>] --fresh-session <peer> --reason <text>
+ahub recovery dispose [<operation-id>] --stop-and-archive --reason <text>
 ```
 
+`--yes` needs `--to`; a dry run without `--to` plans the latest release. A target
+newer than the CLI is handed to that release's coordinator with the same flags,
+and the command is printed on stderr first. The `recovery` commands take the
+operation that holds the machine's lock when the id is left out, and say `no
+recovery operation holds the lock` when none does.
+
 The coordinator commits only once the source is quiet: no turn running, no
 approval pending, no completion check queued or running. It waits up to 10
 minutes and then aborts, leaving the source running; upgrade between long
@@ -1101,6 +1336,10 @@ runs under the deny-default sandbox, and `hub.log` and `ahub doctor` say so.
 `local.bash_network: "direct"` still works until 0.13.0, and both name it while
 it is set.
 
+0.12.22 closes the name `.git` to the local worker's and Pi's commands at any
+depth: `git init`, `git clone` and `git worktree add` no longer run inside the
+sandbox (the full list is under Install and start).
+
 The 0.7.0 transition stages the verified package and runs a retained
 coordinator from the source tree. It accepts a verified protocol-9 source and
 moves to a protocol-10 target. The source journal, queued envelopes, tasks,
@@ -1136,6 +1375,93 @@ Do not run an upgrade with an incompatible active protocol, an unverified
 terminal binding, or an unresolved operation lock. Dry-run performs no package,
 plugin, daemon, or terminal mutation.
 
+### Interactive upgrade
+
+`ahub upgrade` and `ahub restart` are interactive when standard input and output
+are a terminal and neither `--dry-run` nor `--yes` is given. With either flag, or
+through a pipe, the JSON plan, the `ahub: blocker:` lines and the exit codes are
+unchanged. The screens add no way to change an operation: every choice runs one
+of the commands in this section for you, so that command's refusals decide.
+
+The plan screen lists each project with its hub version and, per attached peer,
+its state, how it runs (`TUI` in a terminal the plan bound to its session,
+`headless` as the hub's own process, `own` for a session the hub did not
+launch or whose terminal the plan could not bind) and what happens to it: resumed in a new terminal (the old one is
+named), reconnecting by itself, restarted as a new session, restarted headless,
+or left offline. `in progress:` repeats what the running hub says keeps it from
+being quiet (busy peers, pending approvals, a peer starting, a budget
+transition; a hub of 0.12.22 or newer also names a task command, completion
+checks or Pi tool calls in flight, ready tasks being released and a Pi that has
+not settled); apply waits up to 10 minutes for it. Blockers follow, each with its
+next action. The choices are `[a] apply` (absent while a blocker stands), `[r]
+refresh`, `[k] end agents` (shown when an attached agent can be ended), `[j]`
+the plan as JSON, `[x] reset a project's hub` (shown when the plan has a running
+project) and `[q] quit`.
+
+`[k] end agents` ends attached agents before the upgrade, so they are neither
+waited for nor restored: `t` for the TUI agents, `h` for the headless ones
+(Kimi, Pi in headless mode, the local worker), or their names, written
+`<project id>/<name>` when the plan covers more than one project. After a
+confirmation a TUI agent's terminal is closed, which cuts a turn in progress,
+and a headless agent is stopped by its hub through the `peer_stop` request
+(#278). A delivery that a busy agent had not settled is then held as
+`needs_review`, and its queue stays held until `ahub queue resolve` settles it. Each agent is ended only
+if the lock is free and its hub, its session and its terminal are still the
+ones the plan read; otherwise the screen says why it was left. A hub that does
+not know `peer_stop` (0.12.21 and older) cannot end a headless agent: the
+screen says so, the agent stops with the old hub at the upgrade, and the new
+hub starts it again. A session marked `own` is ended where it runs. The next
+plan shows the ended agents as offline.
+
+After apply the command stays attached and prints each step as the receipt
+records it, and while a source is being prepared, what it still waits for.
+Enter opens the operation's screen while the runner works on. Ctrl+C leaves the
+command, at any prompt and while following, and never stops the runner: the
+`left:` line names the command that comes back to the screen, with the
+operation's own coordinator, because the installed `ahub` is the older release
+until the upgrade ends. A Ctrl+C between `a` and the start of the operation
+starts nothing. A prompt for a reason that is answered with
+Ctrl+C or the end of input ends nothing.
+
+The operation's screen also opens first whenever an operation holds the lock,
+and for bare `ahub recovery`. It shows the phase and step, each project's phase
+and effect receipts and the error, then the choices that `status` lists under
+`next` for the same receipt and live state:
+
+| Key | Runs | Offered when |
+| --- | --- | --- |
+| `r` | `recovery resume`, then follows | resume can get past what is live |
+| `c` | `recovery abort` | no runtime was stopped and no terminal effect or commit request is recorded |
+| `f` | `recovery dispose --fresh-session <peer>`, after a reason | a Codex or Claude restoration failed |
+| `e` | `recovery dispose --stop-and-archive`, after a confirmation and a reason | every unverified project reads as running, stopped or missing |
+| `x` | `e`, then the reset below | as `e` |
+| `w` | follows the runner | a runner holds the operation |
+
+When more than one restoration failed, the second and later peers get
+`f-<peer>`. `s` refreshes the screen, `j` prints the receipt as JSON and `q`
+leaves the operation as it is.
+
+While a runner is only waiting for its source to get quiet (step `prepare`, no
+effect recorded), `c` stops that runner and then aborts. It is offered only for
+a runner whose claim carries a process signature that still matches; a claim
+from a coordinator of 0.12.20 or older has none, so its pid is never signalled,
+and such a runner gives up its wait after 10 minutes. The receipt is read again
+when the key is pressed, and a runner that has moved on is left alone. If it
+records an effect in the instant between that read and the signal, the abort is
+refused and resume is offered instead. Past that step a running operation
+offers only `w`.
+
+A reset is offered from the plan screen for a project whose hub is running, and
+with one question right after an operation was cancelled or ended from its
+screen: it asks for the project (when there are several) and the scope, prints
+the dry run of `ahub reset` or `ahub reset --all`, and applies it with `--yes`
+only after a second confirmation. While an operation holds the lock it refuses,
+as `ahub reset` does.
+
+Ending a peer's running turn is not part of these screens: finish or interrupt
+it in that peer's terminal, answer approvals with `ahub permit` or the console,
+or cancel and upgrade later.
+
 ### A partial operation
 
 `ahub recovery status <operation-id>` reads the receipt without changing it:
@@ -1179,14 +1505,22 @@ next actions").
   get past what is live: not past a replaced or missing source, a source that
   crashed before any commit request, a refusal at staging (the target release
   or the preserved source can never change back), a hub of another control
-  protocol, a target that stopped after it started, or a stopped hub without
-  the operation's restart snapshot.
+  protocol, a target that stopped after it started when the operation's
+  coordinator is older than 0.12.21, or a stopped hub without the operation's
+  restart snapshot.
   A second `resume` while its own hold still stands checks the peers again
   before it closes anything.
-- When a target hub dies after it started (a crash or a reboot), restarting it
-  from its snapshot is not supported in this release (its peers would have to
-  be relaunched against the new hub): `resume` is not offered and the way out
-  is stop-and-archive.
+- When a target hub dies after it started (a crash or a reboot), `resume`
+  starts it again from the operation's snapshot while the project's state
+  directory still holds it: it records the restart (status shows `restarts`),
+  closes the terminals the dead hub's peers were restored into (or finds them
+  gone), and relaunches those peers against the new hub, a session you had
+  accepted instead of the original included. A launcher still running from
+  the dead hub, or a session attached without an id, stops it until you end
+  that session. A target that dies again in the same `resume` stops it with the
+  restart count; look at its `hub.log` before resuming again. Without the
+  snapshot, or for an operation started by a coordinator older than 0.12.21,
+  the way out is stop-and-archive.
 - A Codex conversation comes back only when a rollout file naming its thread
   exists under `sessions/` of the store the restored terminal uses (the
   recorded `CODEX_HOME`, else `~/.codex`). Codex writes that file with the
@@ -1220,7 +1554,13 @@ next actions").
   be read counts as unknown, never as "nothing attached", and so does a launcher
   record file that cannot be read: `resume` then stops without changing any
   receipt; when the record file is what cannot be read, inspect it and move it
-  aside, then `resume`. `resume` is refused while a stop-and-archive is partway. A lifecycle command refused by the lock names
+  aside, then `resume`. A record file that parses but holds a row that cannot
+  be evaluated (a hand edit, a damaged or a future layout) is read the same way
+  for the peer that row may belong to, and `ahub codex`, `ahub claude` and
+  `ahub pi --mode tui` write such rows back unchanged: fix or remove that row,
+  then `resume`. A launcher recorded by 0.12.12 or older is read with its own
+  unpinned signature too, so it counts as running when this command runs in
+  the same time zone and locale as the launcher did. `resume` is refused while a stop-and-archive is partway. A lifecycle command refused by the lock names
   the operation's own `status` command, which lists what to do next.
 - `ahub recovery dispose <operation-id> --fresh-session <peer> --reason <text>`
   applies to a Codex or Claude peer whose restoration failed (Pi is refused: a
@@ -1234,12 +1574,22 @@ next actions").
   <id>` instead of offering it. A hub that never answers (its pid is alive,
   nothing replies): take the `pid` from that project's
   `.agenthub/state/status.json`, check with `ps -p <pid> -o command=` that it
-  is that project's hub daemon, end it by hand, then run `status` again. If
-  that pid now belongs to another process (reused after a reboot), move the
-  stale `status.json` aside instead, then run `status` again; likewise when
-  `next` says `wait: runner <pid>` and that pid is not an `ahub recovery-run`
-  process, move the operation's `~/.agenthub/recovery/<id>.json.runner.db`
-  aside, then run `status` again. A
+  is that project's hub daemon, end it by hand, then run `status` again. From
+  0.12.21 a hub's manifest and registry claim and a runner's record carry the
+  owner's process signature, so a pid that a reboot handed to another process
+  reads as gone by itself. Records written by 0.12.20 or older carry none and
+  are judged by their pid alone: if that pid now belongs to another process,
+  move the stale `status.json` and `hub.pid` aside together (with only one of
+  them gone the other still reads as an owner), then run `status` again;
+  likewise when `next` says `wait: runner <pid>` and that pid is not an `ahub
+  recovery-run` process, move the operation's
+  `~/.agenthub/recovery/<id>.json.runner.db` aside, then run `status` again.
+  An unsigned registry claim of such a hub, whose pid now belongs to another
+  process, still reads as starting, and no command clears it: after `ps -p
+  <pid> -o command=` shows that the process is not an `ahub ... daemon` of that
+  project, clear it with `sqlite3 ~/.agenthub/registry.db "UPDATE projects SET
+  instance_id = NULL, pid = NULL, claimed_at = NULL WHERE id = '<project id>'
+  AND pid = <pid>"` (`ahub projects --json` shows the id). A
   project whose directory is
   gone is only recorded (`ahub doctor --orphans` lists a hub left running
   there). It records its decision before it acts. It releases a source hold of
@@ -1249,7 +1599,13 @@ next actions").
   `restart.json` aside as `restart.abandoned.<hash>.json` and leaves any other
   hub running; only then is the operation recorded `cancelled` and the lock
   released. If it fails partway it keeps the lock, records what it did, and
-  `resume` refuses until you run it again. Queued messages are not delivered
+  `resume` refuses until you run it again. Once a stop-and-archive is recorded,
+  only the command `next` names continues it. On an operation whose
+  coordinator predates these commands it also marks the receipt `schema: 2`,
+  which every older release refuses before it acts: an older global `ahub
+  recovery resume` still prints "scheduled", but its runner exits without
+  touching the operation, and an older `ahub recovery abort` refuses with
+  "unsupported operation receipt". Queued messages are not delivered
   from the archived file; a 0.7.0 or later hub reloads the queues it kept in
   `hub.db` when it starts again. Terminals it closed stay closed: start those
   sessions again by hand. Replacement terminals it opened stay open but have
@@ -1508,8 +1864,8 @@ The hooks return no facts in advisory mode. A native Stop establishes an idle
 boundary; a subsequent PreToolUse marks activity. A delivery acknowledgement or
 task approval does not establish native idle. Restart the daemon and relaunch
 Claude after enabling the sweep so its session receives the hooks. A caller's
-`--settings` still wins: the launcher warns that native idle observation is off,
-and the sweep cannot verify that Claude session between turns.
+`--settings` keeps them: the launcher adds the hub's hooks to the caller's and
+warns only that the status line tee is off.
 
 `auto_reassign: true` explicitly allows an available alternative owner selected
 under the existing routing, role and PII constraints to receive the task at step
@@ -1634,3 +1990,103 @@ Quota pause and task handoff are separate; a context checkpoint neither pauses
 nor hands work over. Continue normally or deliberately restart into a fresh
 session with your chosen checkpoint as preface. No automatic restart or native
 compaction override is performed.
+
+## Pi tool refusals, exit causes and idle automatic restart
+
+Pi hub-task/conductor refusals proven before any task or board effect return
+their original `error:` text and are recorded as completed tool receipts.
+PII screening and its telemetry record may precede the refusal. An uncertain
+post-effect or storage failure remains pending: inspect it before repeating an
+action. Old pending receipts carry no proof of a past refusal and stay pending
+across startup. Conductor assignment and escalation entry checks explicitly opt
+into refusal typing; internal escalation after a saved review remains generic.
+
+Pi exit records show code or signal, turn/tool activity, whether teardown was
+expected, startup status, and the last tool's name without arguments. An
+unobserved native-terminal exit records unknown OS status. The console notice
+gives the next action, usually `ahub pi`, or says why automatic start is held.
+A replacement says the owner stopped for a new Pi owner, including when the
+replacement is refused. Internal failure teardown directs session inspection
+rather than claiming a person requested it. Replacement, failed-start and
+active-turn guidance appears before automatic-start-disabled guidance.
+
+With `pi.auto_start`, an unexpectedly idle headless Pi is resumed on its
+verified persisted session once. The 60-second retry window rearms when the
+attempt settles; a second exit inside that window stays offline. Missing or
+invalid session history has no fresh fallback; inspect it before running
+`ahub pi`. Active turns/tools, failed startup, a superseded owner, native
+terminal exits, a requested stop (`ahub stop pi`), requested shutdown and
+recovery operations suppress this restart. Crash recovery's #66
+recorded-session/fresh-start choices are unchanged.
+
+### Pi/local unanswered approvals
+
+An expired Pi/local request says "approval expired: no person answered". The
+worker must not retry that call; hand the task off or stop. Two consecutive
+unanswered requests end the current turn and produce a console notice. An
+expiry counts only when no person answered a request of that peer while it
+was pending, so a batch of parallel requests does not stop a worker whose
+person is answering; a grant served from the always-allow cache or a permission
+mode is not an answer and does not reset the count. A new turn also resets it. The turn ends
+through the same abort path an execution-budget stop uses: Pi stays attached
+and the sender is told the approval reason. Supported Pi and local settle that
+delivery after reporting the reason, so the next queued message runs without
+an operator release. Pi clears turn-only approval state at settlement; the
+person's idle `!command` remains usable and a later failure keeps its own reason.
+A Pi whose extension cannot abort
+a turn (an older hub extension or a runtime without native abort support) goes
+offline instead. Inspect prior work, restart with `ahub pi`, find the held
+delivery with `ahub queue list --peer pi`, inspect it with `ahub queue show <id>`,
+then explicitly settle it with `ahub queue resolve <id> --action completed|retry|discard --reason "<text>"`.
+A cancelled tool withdraws
+its request immediately; a later `ahub permit` answer is refused and cannot
+create an always-allow grant. Tool requests from a previous turn or an older
+extension missing session/turn identity are refused before execution; restart
+Pi with the current hub extension rather than retrying that old call.
+
+The dashboard shows the number of waiting Pi/local requests and where to
+answer them. It remains deny-only for those private rows. Use the console or
+`ahub tail` to inspect the request, then `ahub permit` to answer it. Recovery
+stops waiting for an approval when that request is withdrawn.
+
+A person's `!command` in the managed Pi TUI runs through the managed sandbox
+without asking for another hub approval. Its reported exit code comes from
+the process, including a nonzero exit; text printed by the command does not
+set its status. This exception requires the current idle TUI owner's reserved
+request and does not apply to model-origin bash calls.
+
+### Stop one headless peer
+
+`ahub stop kimi`, `ahub stop pi` (headless mode), or `ahub stop local` stops
+that hub-owned runtime through its adapter and prints `<peer>: offline`.
+The hub and its other peers keep running. The command is human-only;
+agent shells and tools-role connections cannot use the `peer_stop` request.
+An older hub that does not know the request needs an upgrade before this
+command is available.
+
+Claude, Codex and Pi in TUI mode belong to their native terminals: end them
+in those terminals. The upgrade plan's end-agents action closes a TUI through
+its verified terminal binding and uses `peer_stop` for a headless runtime.
+Unknown and already-offline peers are refused with a next action. A recovery
+operation or shutdown holds stop mutations; finish that operation first.
+A peer's concurrent start, stop or permission-mode change must also finish
+before another lifecycle change is admitted, and an upgrade or restart does not
+prepare the hub while a stop is in progress.
+
+A requested stop does not automatically restart a peer. A person or an authorized
+conductor can explicitly start it again; conductor starts remain permitted. Start it with
+`ahub kimi`, `ahub pi` or `ahub local`. Pending approvals are withdrawn before
+teardown. An in-flight delivery becomes `needs_review`, with a reason naming
+the requested stop; it is neither completed nor automatically replayed.
+Inspect partial effects with `ahub queue list --peer <peer>` and
+`ahub queue show <delivery-id>`, then explicitly choose `completed`, `retry`
+or `discard` with `ahub queue resolve`. Restarting alone does not release
+that held delivery. Work queued behind an idle stopped peer is retained and
+runs when it is started again, subject to its existing pauses and holds. New work
+arriving during teardown stays queued too, even if a console resume runs meanwhile.
+An idle Pi with no message/file yet records its live empty-session proof before
+stopping, so its next start resumes that identity. A Pi stopped after its first
+prompt was accepted but before history was persisted is not treated as empty:
+its delivery is held, and another start refuses until resume history is verified.
+Inspect and settle the held delivery before deliberately replacing that unpersisted
+session by restarting the hub; the stop never silently chooses a fresh session.