@staix/agent-hub 0.12.21 → 0.12.23

package/docs/operations.md

@@ -1,8 +1,25 @@
 # Operations guide
 
-This guide describes ahub 0.12.21 and control protocol 16. Live verification
+This guide describes ahub 0.12.23 and control protocol 16. Live verification
 results and remaining prerequisites are recorded separately in [the smoke ledger](smoke.md).
 
+## Command output
+
+`ahub status`, `board`, `budget`, `doctor`, `projects`, `queue list`, `turns` and `report` print shared console tables and labelled details.
+`status --all` uses the projects table. `queue show`, `models status` and `budget execution status` show labelled fields; add `--json` for the raw fetched document.
+`report --by task` keeps the attribution and coverage sentences with section headings and aligned counters.
+Use `--json` for scripts; it returns the fetched data without color, and `status --json` keeps its existing document.
+In `projects --json` and `status --all --json`, an unavailable hub that explicitly refuses an inspection for SQLite contention also carries `failure: { "code": "database_busy" }`; it is additive metadata alongside the neutral `error` sentence.
+Text columns are a human interface, not a scripting contract. An identifier accepted as a command argument is whole wherever its available column can hold its own length:
+turn ids, queue ids, project ids, execution budget ids and ids in labelled views. Only status's informational settlement and hold ids are shortened;
+`status --full` keeps those whole too. Identifiers in suggested commands are always whole.
+
+`--color=auto|always|never` uses the console palette. Auto colors only a TTY with `TERM` other than `dumb`
+and empty `NO_COLOR`; pipes and redirects stay plain. Color never replaces a state or level word.
+TTY width is its columns (at least 80). A pipe uses a positive integer `COLUMNS` when supplied, otherwise rows
+are complete and unwrapped. Long fields wrap within their column; when whole metadata words and the readable flexible-column minimum do not fit, table bands retain every field. Each band carries the row id; when it cannot share a line with the metadata, a per-row id banner links them. Labelled two-column views keep each label beside its value, wrapping the value instead of separating labels and values into bands. In `turns`, the time and state cells wrap at word boundaries only when the file list needs the room and the listing gets shorter for it. When `projects` bands, the task list stays with the state columns in the first band and the second band is always PROJECT and ROOT. Console children receive the console's width
+minus its four-column indent and remain uncolored.
+
 ## Command help
 
 `ahub help`, `ahub --help` and `ahub -h` print the commands grouped by section;
@@ -46,6 +63,15 @@ a three-line footer, separated by ASCII `-` rules; the stream has one rule above
 its footer. The active tab is bracketed, as in `[2 Approvals]`, so it reads
 without color. The Approvals tab counts pending requests and the Queue tab
 counts `needs_review` deliveries, in the attention and failure tones.
+Tabs change with the digits 1 to 5 or with Left and Right (wrapping at both
+ends), never while a command is being edited or a choice is open; under a
+confirmation an arrow dismisses it like any key but y, without moving a tab. A pasted
+chunk can only edit the command line, but a read that holds nothing but arrow
+sequences is that many key presses, so a held key repeats its tab move. In the Events
+tab a row's header line is tokenized and coloured exactly as the same event in
+the stream; only the header is tab-flattened before tokenizing. Body lines retain
+the previous rendering and stay plain, as does command output; colour never changes
+what the panel's text says.
 
 Peers, Approvals, Tasks and Queue are tables with a header row. A column is as
 wide as its widest cell, up to a third of the terminal; the last column takes
@@ -78,7 +104,7 @@ prompts, are quoted unless they are a single word starting with a letter, so
 punctuation, a newline or a bare number in them cannot imitate the console's own
 structure. Under the `?` key table only `?`, Escape and `q` act.
 
-Arrow keys or j/k move, Enter opens a detail of labeled fields (relative times,
+Up/Down arrows or j/k move, Enter opens a detail of labeled fields (relative times,
 never JSON), Escape returns, and `?` shows the keys grouped by panel; in the
 stream, `?` prints that table and `v` prints the selected request as its push
 showed it. The footer's last line lists only the keys that act in the current
@@ -199,10 +225,12 @@ Launch that peer with `ahub claude` for channel pushes, or select `codex` in
 `roles` and launch `ahub codex`. The conductor splits work into owned tasks,
 observes `hub_status`, moves stalled work and obtains review before reporting
 results and open decisions. It does not implement the tasks it handed out.
-`hub_peer_start` starts local, Kimi or headless Pi; requests for native TUIs
-return the `ahub` command for the person to run, after validating it through
-the shared launcher planner. The wrapper plans again at launch when native
-endpoints are available. `hub_peer_hold` and `hub_peer_release`
+`hub_peer_start` starts a peer in the start mode a person set for it (see
+"Start modes"): local, Kimi and a headless Pi or Codex start at once; for a
+TUI peer the hub opens a terminal running the fixed `ahub <peer>` command,
+after validating it through the shared launcher planner, and when it cannot
+open one the call fails with that command for the person to run. The
+conductor cannot choose the mode. `hub_peer_hold` and `hub_peer_release`
 manage only holds placed by that conductor. Assignment also requires `assign`
 when the conductor has an explicit capabilities list.
 
@@ -213,7 +241,8 @@ final response in the native TUI. Broadcasting implementation instructions can
 cause an otherwise unassigned owner to claim duplicate work.
 
 The person answers approvals in the console, resolves `needs_review` deliveries
-with `ahub queue resolve`, and handles budget overrides and hub lifecycle.
+with `ahub queue resolve`, stops a headless peer with `ahub stop`, and handles
+budget overrides and hub lifecycle.
 Running these commands from an agent shell is refused; the CLI retains the
 agent's identity even when invoked through a shell tool.
 
@@ -286,6 +315,19 @@ project and a temp dir of their own (`TMPDIR`, made for each command and
 removed when it ends; one left by a hub crash, named `ahub-cmd-*`, goes with
 the OS temp cleanup), and nothing else; the shared temp dirs are closed.
 
+Some names stay closed to those commands, and to Pi's, wherever they may
+otherwise write: anything named `.git` at any depth, the `config` and `hooks`
+directly under a `.git`, `commondir` below it, and `.agenthub` anywhere in the
+project. A submodule's git directory under `.git/modules/` is not covered yet
+(#281). A command can therefore not create, rename or remove a `.git`:
+`git init`, `git clone` and `git worktree add` do not run inside the sandbox,
+and neither do `git worktree remove`, `git worktree prune` of a stale record,
+`git clean -ffd` over a nested repository or `rm -rf` of a directory that
+holds a `.git`. `git add`, `commit`, `stash`, `checkout` and `gc` work, and
+`.gitignore` and `.github/` stay writable; a commit that your git configuration
+signs with a key under `~/.ssh` or `~/.gnupg` fails, because the sandbox cannot
+read those directories. Run a refused command yourself, in a terminal.
+
 Network is off unless `local.bash_network` says otherwise. With `true` it goes
 only through the hub's egress proxy on a loopback port (0.11): commands get
 `HTTPS_PROXY` and the other proxy variables, the proxy opens HTTPS (`CONNECT`,
@@ -337,7 +379,7 @@ Managed launchers attach native peers to the project daemon:
 ahub claude
 ahub codex
 ahub kimi
-ahub pi --mode headless --backend auto
+ahub pi                 # its TUI; --headless for the background peer
 ahub local
 ahub local --model <served-model-id>
 ```
@@ -366,7 +408,19 @@ A delivery in `needs_review` holds later deliveries to that peer. The console an
 the hold once, and status, task assignment and route explanations name its delivery id.
 Inspect it with `ahub queue show <id>` and explicitly choose `completed`, `retry` or
 `discard` through `ahub queue resolve`. Reassignment or approval of its task never
-resolves an uncertain delivery automatically.
+resolves an uncertain delivery automatically. The same rule holds on every path that
+assigns a task (#297): once the move is saved on the board, a failed delivery prints
+exactly one console notice ("the move is saved... delivery is not confirmed"), printed
+where the error is thrown, and the caller (a peer in its turn, or the CLI) still gets
+its error. Two exceptions, by construction: a move superseded while the assignment was being prepared, meaning a later history entry that moves or settles the task (another assignment, an unassign, the owner's accept, done, approval, close; the hub's quiet bookkeeping such as a late check result or a reviewer change does not supersede), is
+the caller's error as before, with no notice naming either owner; and a claim is its own hand-over, so its
+accepted write and line stay outside the delivery rule (its overlap and routing announcements to earlier
+owners are deliveries and share it). The repair is always
+to look for a queued or needs_review delivery with
+`ahub queue list --peer <owner>` (then `ahub queue show <id>`, or the accepted entry in
+`ahub task show <id>` if the journal itself is down) and then send the assignment again
+with `ahub task assign <id> <owner>`. Nothing resends automatically: a blind resend can
+duplicate a delivery that actually left.
 
 Reviewer candidates follow `[classes.review].peers`, then attached peers with a
 `reviewer` role in `roles`. The owner cannot review its own task. If none is available,
@@ -630,7 +684,7 @@ ahub undo <turn> --yes --context  # Codex's latest turn: also drop it from Codex
   has not removed yet included, and a PII file left in the project is
   snapshotted by later turns like any other file. Keep PII files in an ignored
   directory.
-- A turn the hub stopped in the middle of shows as `(no end snapshot)` and
+- A turn the hub stopped in the middle of shows the state `no end snapshot` and
   cannot be undone.
 - Claude's turns are not recorded: its channel shows the hub no turn boundary.
   Claude Code's own checkpoints cover its edits, though not its shell commands.
@@ -881,12 +935,36 @@ Other tools retain native decisions, which may already allow a harmless shell.
 Permission-only PreToolUse never marks Claude busy; former observation hooks
 retain their native-turn role. Settings are passed as a 0600 state file retained
 until that native launch exits. Live/unknown previous launchers keep their files;
-verified dead wrapper and native identities permit crash cleanup. Without
+verified dead wrapper and native identities permit crash cleanup. Managed direct
+binary launches record their native PID while running; crash cleanup also requires
+that PID to be absent from the process table. Scripts/interpreters and unavailable
+process evidence retain the settings file. Every launch atomically replaces
+`claude-launch.json` with its current managed hook/session identity. Native PID
+proof is separate in a private `claude-native-<binding-hash>.json` file, so an old
+callback cannot overwrite a newer launch record. Only that native proof uses
+`claude-launch.lock`, published with its signed owner already written. It waits
+up to 250 ms for a verified live owner. An empty, unsigned or unverifiable lock
+is retained and disables native identity/crash cleanup for this launch; the
+warning names the path and what is off. Managed hooks and session observations
+still use the newly published launch record. Inspect the named lock and verify
+its owner has ended before removing it; a signed dead lock is recovered. Without
 separate native identity proof, old settings remain for manual inspection. Settings carry
 no inline caller values in argv. Status reports an unverified Claude hook as
 unverified. Unattended native sessions cannot take a runtime mode; restart
 without that native flag. Codex must be behind the hub proxy, and returning to
-ask requires a known native approval baseline. An offline or absent peer accepts
+ask while attached requires a known native approval baseline. If unavailable,
+close the TUI, then run `ahub permission codex ask` before starting `ahub codex`.
+For a known baseline, a detached proxy keeps restoration owed per thread and sends
+that policy once on the resumed thread's next accepted turn. Visiting another thread
+does not clear the debt. The restoring turn uses that kept baseline even if the
+TUI echoes the resume response's sticky override, whether ask was selected before
+or after that resume. Rejected turns retain it. The protection covers that one
+restoring turn. A later TUI turn that repeats the policy looks exactly like a
+person's explicit native policy choice and is passed through; the proxy cannot
+resolve that ambiguity. An explicit choice on the restoring turn is delayed
+until the following turn. A baseline that was never reported
+is explained in the CLI reply, and that old thread keeps its last policy.
+An offline or absent peer accepts
 `ahub permission <peer> ask` to clear its hub choice. Codex --unattended launch
 is refused under a non-ask choice; clear it with `ahub permission codex ask`
 first. A pending never-ask default cannot be confirmed for an unattended launch.
@@ -918,6 +996,130 @@ permission_mode (peer/from/to) with no tool arguments under events schema 1.
 The optional control requests preserve PROTOCOL; an older hub answers unknown
 and the CLI advises upgrading.
 
+## Settings from the dashboard and the terminal
+
+The dashboard's Settings section and `ahub settings` show and change the same
+closed list of settings (issue #269). Run them from a plain terminal:
+
+```bash
+ahub settings                      # every setting: value, source file, when it applies
+ahub settings get permission.kimi
+ahub settings set routing.classes.implement.peers codex,kimi,pi --preview
+ahub settings set routing.classes.implement.peers codex,kimi,pi
+ahub settings set permission_modes.kimi ask-when-needed
+ahub settings set research.enabled inherit   # remove the machine-local value
+ahub settings undo                 # put back the file the last write changed
+ahub ui --settings                 # dashboard with a 15-minute settings session
+```
+
+| Group | Keys | Stored in | Applies |
+| --- | --- | --- | --- |
+| Permissions | `permission.<peer>` (the mode now) | the running hub only | at once |
+| Permissions | `permission_modes.<peer>` (the mode at hub start) | `config.local.json` | next hub start |
+| Start | `peers.pi.start_mode`, `peers.codex.start_mode` (`tui` or `headless`) | `config.local.json` | that peer's next start |
+| Start | `pi.auto_start` | `config.local.json` | next hub start |
+| Routing | `routing.stay_switch`, `routing.classes.<class>.peers`, `.escalate_to`, `.route`, `.pi_backend` | `routing.local.toml` | at once |
+| Switches | `research.enabled`, `approvals.notify`, `snapshots.enabled`, `coordination` | `config.local.json` | next hub start |
+
+- Writes go only to `.agenthub/config.local.json` and `.agenthub/routing.local.toml`,
+  the machine-local files (`ahub init` ignores both in git). The shared
+  `config.json` and the hand-written `routing.toml` are never rewritten, and a
+  machine-local file that is committed to git is refused. Each write is read by
+  the hub's own loader first and replaces the file atomically.
+- `routing.local.toml` overrides `routing.toml` key by key and may hold only the
+  routing keys in the table. `ahub route explain` and the Settings rows name the
+  file each value comes from. `--preview`, and the dashboard's Preview button,
+  show `ahub route explain` for the open tasks before and after, or what a
+  permission mode grants, and save nothing.
+- A row shows the value in force, its source file, when it applies, and for a
+  value read at hub start what the next start will read (`Pending restart`).
+- `ahub settings undo` (the dashboard's "Undo last write") puts back the one
+  previous version of the file the last write changed. It refuses when that
+  file was edited since, and when it would put a stored never-ask back (set it
+  again with its confirmation instead). A mode changed for the running hub is
+  not a file: set it again instead. Setting a value it already has, or
+  removing one that is not set, writes nothing. A project with no
+  `.agenthub/config.json` gets no first config file from a settings write
+  (any config file turns the project defaults on): run `ahub init` first.
+- A project initialised before 0.12.23 has no `.gitignore` line for
+  `.agenthub/routing.local.toml`: run `ahub init` again, or add it. A
+  `permission_modes` value is read only from a file git confirms nobody
+  committed (config trust); where the hub will not read what was written, the
+  reply and the Settings row say so.
+- Authority: a dashboard opened with `ahub ui` reads settings, lowers a mode to
+  `ask`, turns `pi.auto_start` off, sets a start mode back to `tui` and changes
+  the Switches group. Everything
+  else needs `ahub ui --settings`, whose session may raise for 15 minutes, or
+  `ahub settings` in a terminal. `never-ask` needs the peer id typed in the
+  dashboard, or `--yes` in the terminal, and a stored never-ask default still
+  waits for `y` in `ahub console` at each hub start. Tool approvals stay
+  deny-only for Pi and local in every session.
+- Commands and binaries, credential files, gateway URLs and the local worker's
+  reach are never editable here: edit `.agenthub/config.local.json` yourself.
+- Every change is announced in `hub.log` and open consoles with its source and
+  recorded as a `settings` event. `ahub settings` needs a running hub; an older
+  hub answers the request as unknown and the CLI advises upgrading. The unified
+  dashboard (`ahub ui --all`) acts as an ordinary session.
+
+## Start modes: TUI by default, headless as opt-in
+
+A peer that has a TUI the hub attaches to starts in it by default (issue #269).
+
+| Peer | Default | Headless |
+| --- | --- | --- |
+| pi | `tui` | `ahub pi --headless` (or `--mode headless`), or `peers.pi.start_mode: headless` |
+| codex | `tui` (`ahub codex`) | `peers.codex.start_mode: headless`: a hub-made start runs the app-server only |
+| claude | `tui` (`ahub claude`) | none |
+| kimi | headless (ACP) | it has no TUI the hub attaches to |
+| local | headless (hub-native) | not applicable |
+
+- **You type the command:** `ahub pi` opens Pi's TUI in that terminal.
+  `--headless` and the start mode setting are the two ways to ask for the
+  background peer. `ahub settings set peers.pi.start_mode headless` writes the
+  setting; it applies at Pi's next start.
+- **The hub starts the peer** (the dashboard's Start control, `pi` typed in
+  `ahub console`, the conductor's `hub_peer_start`, `pi.auto_start`): it
+  follows the peer's start mode. For `tui` it opens a terminal running the
+  fixed command `ahub <peer>`, through a terminal provider:
+  1. `terminal.open` in `.agenthub/config.local.json`, an argv in which one
+     element holds `{command}` (the quoted command line to run); `{title}` and
+     `{cwd}` are replaced too. Examples:
+     `["tmux", "new-window", "-n", "{title}", "{command}"]`,
+     `["wezterm", "start", "--cwd", "{cwd}", "--", "sh", "-c", "{command}"]`.
+     Only a config file nobody committed may set it, and it is not a setting
+     the dashboard or `ahub settings` can write. It is read at hub start.
+     `{command}` is quoted for a POSIX shell; `{title}` and `{cwd}` are raw,
+     so give each an argv element of its own.
+  2. Orca, when the hub runs in an Orca worktree or an earlier `ahub <peer>`
+     launch of this project recorded one (the path recovery uses).
+- **No provider:** the start is refused with the command to run by hand. It is
+  never downgraded to headless. `ahub settings` lists each start mode, and the
+  dashboard's Start control says which mode a start will use and where its
+  terminal comes from.
+- **Dashboard Start:** the control is locked while its request is in flight. A confirmed answer unlocks it immediately, so a retry reaches the hub's own 30-second terminal reopen guard and shows any known refusal. The unified dashboard waits 35 seconds, covering a terminal provider's 30-second cap plus a 5-second response margin; a slower headless start can still end unconfirmed while the hub continues. Check the Peers panel before retrying. An unconfirmed unified-dashboard start holds that peer's control for the shared 30-second reopen bound, or until its `starts` snapshot confirms attachment, across redraws and project switches within the same hub instance; the row states the retry bound. The project-local dashboard keeps its usual confirmed-answer behavior. Other forwarded actions keep their 10-second wait; the manager opens no settings session.
+- The conductor cannot choose the mode: `hub_peer_start` takes the peer's own.
+- Recovery restores each peer in the mode its recorded session had. After an
+  unplanned stop, a Pi that ran in a terminal is reported with its command and
+  is not replaced by a headless one unless headless is its start mode.
+
+### Migration from 0.12.22 and earlier
+
+Pi's default changes from headless to its TUI.
+
+- `ahub pi` in a terminal now opens the TUI. For the background peer run
+  `ahub pi --headless`.
+- A project with `"pi": { "auto_start": true }` used to get a headless Pi at
+  every hub start. To keep that, set the start mode once:
+  `ahub settings set peers.pi.start_mode headless` (or add
+  `"peers": { "pi": { "start_mode": "headless" } }` to
+  `.agenthub/config.local.json`). Without it the hub opens Pi's TUI in a
+  terminal at start, or, with no terminal provider, says
+  `pi.auto_start did not start Pi` with the command to run.
+- Scripts and the console's `pi` command: without a terminal, `ahub pi` asks
+  the hub to open one. Add `--headless` where a headless Pi is meant.
+- A conductor's `hub_peer_start` no longer takes a mode. A call that passes one
+  is refused unless it repeats the peer's start mode.
+
 ## Approvals and pauses
 
 Inspect permission requests in the terminal:
@@ -936,9 +1138,11 @@ ahub permit <request-id> allow
 
 Use the exact option shown by `ahub tail`; do not approve an unresolved or
 unexpected request. On macOS a waiting request also raises a desktop
-notification that names the peer and, for Kimi and the local worker, the tool,
+notification that names the peer and, for Kimi, Pi and the local worker, the tool,
 never what it would run. An unanswered request is cancelled after `approvals.timeout_s` (default
 120, 30 to 3600) in `.agenthub/config.json`, and the console and log say so.
+Pi and the local worker are told that nobody answered, and two such requests in
+a row end their turn ([Pi/local unanswered approvals](#pilocal-unanswered-approvals)).
 Set `approvals.notify` to `false` to turn notifications off; a hub started
 without a project config file raises none. While Kimi waits for an answer its
 turn is kept alive, so a timeout longer than the inactivity watchdog does not
@@ -955,6 +1159,13 @@ A budget pause remains authoritative until the budget command explicitly
 overrides it or the window resets. Check `ahub status` and `ahub board` after
 a pause or handoff.
 
+Kimi Code CLI 2.1.1's observed 5-hour usage-limit refusal creates a hard budget reading (`kimi ACP usage-limit refusal`). `ahub status` shows a budget pause and `ahub budget` shows its source and reset estimate. The refusal gives no reset timestamp, so the hub estimates the reset at five hours after it was observed and resumes on the next budget tick after the existing one-minute grace. With `budget.wait_max_min` below that estimate, the existing budget relay automatically hands open tasks to an attached eligible peer; routing skips Kimi while paused. No checkpoint is requested from an exhausted agent.
+`ahub budget resume kimi` overrides the reading so the provider can be tried early. A new provider refusal ends that override and pauses it again; a pending soft checkpoint cannot postpone this hard pause.
+During one hub run, an early override followed by the same window's refusal keeps its earlier future reset bound; a provider's explicit reset remains authoritative. A restored pause without a per-window reading cannot identify that earlier window, so its next refusal starts a new estimate.
+Unread quota resume notices are coalesced without losing their public task moves; the next notice lists moves accumulated across those pauses. An older notice without structured move metadata is kept as public text inside that notice.
+
+A refused prompt with no acceptance or activity is requeued without a review hold. At the estimated reset it is tried again; another quota refusal pauses it again without consuming the poison-envelope retry count. Work that already started remains held for review even while its open tasks move to other peers. After the quota reset its queue stays held: inspect `ahub queue show <delivery-id>` and the associated task, then choose `ahub queue resolve <delivery-id> --action completed|retry|discard --reason "<text>"`. Do not retry work that another owner already took over without inspecting it. If the hub cannot confirm the pause, the refusal is also held instead of spending retries. Other errors keep their ordinary review behavior. Only the observed Kimi 5-hour text is recognised; weekly/monthly limits and other ACP vendors, including Qwen, remain unverified. The recognised text and its provenance live in `src/adapters/acp-quota.ts`.
+
 Quota also shapes routing and handoffs:
 
 - Among peers a task could go to, those with quota readings are ordered by
@@ -1012,8 +1223,8 @@ Only `ahub claude` starts Claude Code with the development-channel flag. A
 Claude session started another way (plain `claude`, an IDE) keeps the hub tools
 but cannot show pushes, so it attaches tools-only (protocol 16, issue #205):
 
-- Its messages stay queued. `ahub status` shows `queued N` and `tools-only:
-  messages wait for hub_inbox; for pushes restart Claude with ahub claude`; the
+- Its messages stay queued. `ahub status` shows the queue count in `Q` and a `tools-only` detail:
+  `messages wait for hub_inbox; for pushes restart Claude with ahub claude`; the
   console footer shows `tools-only: ahub claude` and the dashboard shows the
   status line.
 - The session reads them with `hub_inbox`, at most ten messages at a time plus
@@ -1044,11 +1255,15 @@ The daemon waits for shutdown work that must survive termination. Afterward,
 run `ahub status` or `ahub up` and read the project state before restarting
 peers. A stopped daemon does not delete task records or the durable journal.
 Do not kill a native terminal by PID or start a replacement while ownership is
-uncertain.
+uncertain. To stop one headless peer and keep the hub running, use
+`ahub stop <peer>` ([Stop one headless peer](#stop-one-headless-peer)).
 
 Shutdown is bounded: once it begins, a daemon that cannot finish within
 15 seconds exits anyway, and a peer that refuses to stop is logged rather than
-allowed to block state cleanup. A native Pi TUI owner that does not exit
+allowed to block state cleanup. A managed Pi tool that does not settle within
+the stop grace does not block the teardown either: the stop goes on
+(shutdown, owner teardown, process stop) and reports the failure at the end.
+A native Pi TUI owner that does not exit
 gracefully is terminated by its verified process identity (never a bare PID),
 so a restarted hub never launches beside a survivor. A hub whose project root
 or state directory was deleted stops itself within about 10 seconds; the
@@ -1169,23 +1384,38 @@ Rows without a live process are stale registrations; forget them with
 
 ## Upgrade and crash recovery
 
+On a terminal, one command reviews, applies and follows an upgrade:
+
+```bash
+ahub upgrade
+```
+
+Without `--to` the target is the registry's latest release. When that release is
+newer than the installed CLI, the CLI names the release's own coordinator
+(`bun x --package @staix/agent-hub@<version> ahub upgrade --to <version>`), asks,
+and hands over to it, so the version is never typed twice and an older CLI never
+coordinates. From an installed CLI older than 0.12.22, start it once by hand:
+`bunx --package @staix/agent-hub@<version> ahub upgrade`. The screens are
+described under [Interactive upgrade](#interactive-upgrade); the rest of this
+section is what they run.
+
 Upgrade running projects with the target release's own coordinator. It accepts
 a running source on control protocol 9 (0.6.x), 10 (0.7.0 through 0.12.0),
-11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 and 0.12.21), and only
+11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 through 0.12.23), and only
 a target on its own protocol, so the target's coordinator fits every supported
 source and carries every recovery fix released up to it. Protocol 8 and older
 (0.5.x and earlier) are refused as `manual-bootstrap-required`. Run from the
 project directory, without replacing the global CLI first:
 
 ```bash
-bunx --package @staix/agent-hub@0.12.21 ahub upgrade --to 0.12.21 --dry-run
-bunx --package @staix/agent-hub@0.12.21 ahub upgrade --to 0.12.21 --yes
+bunx --package @staix/agent-hub@0.12.23 ahub upgrade --to 0.12.23 --dry-run
+bunx --package @staix/agent-hub@0.12.23 ahub upgrade --to 0.12.23 --yes
 ```
 
 | Running now | Coordinator to use |
 | --- | --- |
 | 0.6.x (protocol 9) | the target's, through `bunx` as above |
-| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 and 0.12.21 (protocol 16) | the target's, through `bunx` as above |
+| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 through 0.12.23 (protocol 16) | the target's, through `bunx` as above |
 | any supported source, with the installed CLI already at the target | `ahub upgrade` below, which is the same coordinator |
 | 0.5.x or earlier (protocol 8 and older) | not supported: bootstrap by hand with the matching CLI |
 
@@ -1217,21 +1447,27 @@ projects first:
 
 ```bash
 ahub restart --dry-run
-ahub upgrade --to 0.12.21 --dry-run
+ahub upgrade --to 0.12.23 --dry-run
 ```
 
 Apply only after reviewing the plan:
 
 ```bash
 ahub restart --yes
-ahub upgrade --to 0.12.21 --yes
-ahub recovery status <operation-id>
-ahub recovery resume <operation-id>
-ahub recovery abort <operation-id>
-ahub recovery dispose <operation-id> --fresh-session <peer> --reason <text>
-ahub recovery dispose <operation-id> --stop-and-archive --reason <text>
+ahub upgrade --to 0.12.23 --yes
+ahub recovery status [<operation-id>]
+ahub recovery resume [<operation-id>]
+ahub recovery abort [<operation-id>]
+ahub recovery dispose [<operation-id>] --fresh-session <peer> --reason <text>
+ahub recovery dispose [<operation-id>] --stop-and-archive --reason <text>
 ```
 
+`--yes` needs `--to`; a dry run without `--to` plans the latest release. A target
+newer than the CLI is handed to that release's coordinator with the same flags,
+and the command is printed on stderr first. The `recovery` commands take the
+operation that holds the machine's lock when the id is left out, and say `no
+recovery operation holds the lock` when none does.
+
 The coordinator commits only once the source is quiet: no turn running, no
 approval pending, no completion check queued or running. It waits up to 10
 minutes and then aborts, leaving the source running; upgrade between long
@@ -1271,6 +1507,10 @@ runs under the deny-default sandbox, and `hub.log` and `ahub doctor` say so.
 `local.bash_network: "direct"` still works until 0.13.0, and both name it while
 it is set.
 
+0.12.22 closes the name `.git` to the local worker's and Pi's commands at any
+depth: `git init`, `git clone` and `git worktree add` no longer run inside the
+sandbox (the full list is under Install and start).
+
 The 0.7.0 transition stages the verified package and runs a retained
 coordinator from the source tree. It accepts a verified protocol-9 source and
 moves to a protocol-10 target. The source journal, queued envelopes, tasks,
@@ -1306,6 +1546,93 @@ Do not run an upgrade with an incompatible active protocol, an unverified
 terminal binding, or an unresolved operation lock. Dry-run performs no package,
 plugin, daemon, or terminal mutation.
 
+### Interactive upgrade
+
+`ahub upgrade` and `ahub restart` are interactive when standard input and output
+are a terminal and neither `--dry-run` nor `--yes` is given. With either flag, or
+through a pipe, the JSON plan, the `ahub: blocker:` lines and the exit codes are
+unchanged. The screens add no way to change an operation: every choice runs one
+of the commands in this section for you, so that command's refusals decide.
+
+The plan screen lists each project with its hub version and, per attached peer,
+its state, how it runs (`TUI` in a terminal the plan bound to its session,
+`headless` as the hub's own process, `own` for a session the hub did not
+launch or whose terminal the plan could not bind) and what happens to it: resumed in a new terminal (the old one is
+named), reconnecting by itself, restarted as a new session, restarted headless,
+or left offline. `in progress:` repeats what the running hub says keeps it from
+being quiet (busy peers, pending approvals, a peer starting, a budget
+transition; a hub of 0.12.22 or newer also names a task command, completion
+checks or Pi tool calls in flight, ready tasks being released and a Pi that has
+not settled); apply waits up to 10 minutes for it. Blockers follow, each with its
+next action. The choices are `[a] apply` (absent while a blocker stands), `[r]
+refresh`, `[k] end agents` (shown when an attached agent can be ended), `[j]`
+the plan as JSON, `[x] reset a project's hub` (shown when the plan has a running
+project) and `[q] quit`.
+
+`[k] end agents` ends attached agents before the upgrade, so they are neither
+waited for nor restored: `t` for the TUI agents, `h` for the headless ones
+(Kimi, Pi in headless mode, the local worker), or their names, written
+`<project id>/<name>` when the plan covers more than one project. After a
+confirmation a TUI agent's terminal is closed, which cuts a turn in progress,
+and a headless agent is stopped by its hub through the `peer_stop` request
+(#278). A delivery that a busy agent had not settled is then held as
+`needs_review`, and its queue stays held until `ahub queue resolve` settles it. Each agent is ended only
+if the lock is free and its hub, its session and its terminal are still the
+ones the plan read; otherwise the screen says why it was left. A hub that does
+not know `peer_stop` (0.12.21 and older) cannot end a headless agent: the
+screen says so, the agent stops with the old hub at the upgrade, and the new
+hub starts it again. A session marked `own` is ended where it runs. The next
+plan shows the ended agents as offline.
+
+After apply the command stays attached and prints each step as the receipt
+records it, and while a source is being prepared, what it still waits for.
+Enter opens the operation's screen while the runner works on. Ctrl+C leaves the
+command, at any prompt and while following, and never stops the runner: the
+`left:` line names the command that comes back to the screen, with the
+operation's own coordinator, because the installed `ahub` is the older release
+until the upgrade ends. A Ctrl+C between `a` and the start of the operation
+starts nothing. A prompt for a reason that is answered with
+Ctrl+C or the end of input ends nothing.
+
+The operation's screen also opens first whenever an operation holds the lock,
+and for bare `ahub recovery`. It shows the phase and step, each project's phase
+and effect receipts and the error, then the choices that `status` lists under
+`next` for the same receipt and live state:
+
+| Key | Runs | Offered when |
+| --- | --- | --- |
+| `r` | `recovery resume`, then follows | resume can get past what is live |
+| `c` | `recovery abort` | no runtime was stopped and no terminal effect or commit request is recorded |
+| `f` | `recovery dispose --fresh-session <peer>`, after a reason | a Codex or Claude restoration failed |
+| `e` | `recovery dispose --stop-and-archive`, after a confirmation and a reason | every unverified project reads as running, stopped or missing |
+| `x` | `e`, then the reset below | as `e` |
+| `w` | follows the runner | a runner holds the operation |
+
+When more than one restoration failed, the second and later peers get
+`f-<peer>`. `s` refreshes the screen, `j` prints the receipt as JSON and `q`
+leaves the operation as it is.
+
+While a runner is only waiting for its source to get quiet (step `prepare`, no
+effect recorded), `c` stops that runner and then aborts. It is offered only for
+a runner whose claim carries a process signature that still matches; a claim
+from a coordinator of 0.12.20 or older has none, so its pid is never signalled,
+and such a runner gives up its wait after 10 minutes. The receipt is read again
+when the key is pressed, and a runner that has moved on is left alone. If it
+records an effect in the instant between that read and the signal, the abort is
+refused and resume is offered instead. Past that step a running operation
+offers only `w`.
+
+A reset is offered from the plan screen for a project whose hub is running, and
+with one question right after an operation was cancelled or ended from its
+screen: it asks for the project (when there are several) and the scope, prints
+the dry run of `ahub reset` or `ahub reset --all`, and applies it with `--yes`
+only after a second confirmation. While an operation holds the lock it refuses,
+as `ahub reset` does.
+
+Ending a peer's running turn is not part of these screens: finish or interrupt
+it in that peer's terminal, answer approvals with `ahub permit` or the console,
+or cancel and upgrade later.
+
 ### A partial operation
 
 `ahub recovery status <operation-id>` reads the receipt without changing it:
@@ -1354,6 +1681,28 @@ next actions").
   restart snapshot.
   A second `resume` while its own hold still stands checks the peers again
   before it closes anything.
+- If a restart stops with `database_busy`, the receipt stays `blocked` and
+  carries `failure: { code: "database_busy", retryable: true, site: [...] }`.
+  Let the other connection settle, inspect `ahub recovery status <id>` and
+  use its offered `ahub recovery resume <id>`. Resume rechecks the source and
+  reconciles a commit that may already have been requested. A busy read while
+  that committed source is stopping stays inside the existing shutdown wait.
+  A stop-and-archive that meets contention instead says to retry that
+  disposition; resume remains refused while abandonment is partway. Successful
+  abort, fresh-session choice and disposal clear the earlier failure annotation.
+  Do not delete database files or start a second operation to bypass the lock.
+  A failed installed-layout CI restart retains bounded, redacted fixture logs,
+  receipt fields (including `commitSent` and failure frames) and file-mode
+  metadata as a per-OS artifact.
+
+  The historical CI receipt (38061057253, attempt 1) kept no throw site or
+  hub log. At its main, status/inspect RPC failures were swallowed and the
+  synchronous catch sent a fixed sentence, so its raw SQLite text could only
+  have come from the commit rejection or the runner's own `registry.db` read.
+  "Resume | wait until the hub settles", without stop-and-archive, means its
+  next inspection saw neither running nor stopped/missing. Which of the two
+  paths, which statement and which holder remain unknown. The controlled
+  rollback-copy reproduction is separate evidence, not that historical cause.
 - When a target hub dies after it started (a crash or a reboot), `resume`
   starts it again from the operation's snapshot while the project's state
   directory still holds it: it records the restart (status shows `restarts`),
@@ -1492,8 +1841,9 @@ crash of one is not reported this way.
   spends no cloud quota), and on a fresh session if that fails, keeping the
   recorded backend and model; the report says which. Malformed records in
   `sessions.json` are skipped. A Pi that ran in a terminal (`--mode tui`) is never started on its
-  recorded session by the hub; the report gives the command, and with
-  `pi.auto_start` a fresh headless Pi starts instead.
+  recorded session by the hub; the report gives the command. With
+  `pi.auto_start` a fresh headless Pi starts instead only when Pi's start mode
+  is headless; with the default (tui) nothing is started in its place.
 - Codex's app-server died with the hub; run `ahub codex` again. Claude Code's
   plugin reconnects by itself while that session is open.
 
@@ -1620,7 +1970,7 @@ whose MLX slot is still busy after that wait goes to `dgx/fast` first (`source:
 so does an enforced tool loop pinned to it (`source: "pin"`); without it nothing moves for load. `dgx/fast`
 never goes first while it is cooling down, or while its own last dispatch failed in any way,
 an error status or a failed stream included, until it succeeds or 30 s pass (`ahub status`:
-`last dispatch failed, no load moves until ...`). No load move or pin happens while the request
+a `dispatch held` detail with a relative time and `no load moves until then`). No load move or pin happens while the request
 runs under an execution budget, and enforced, a load move happens only at a user turn. A moved
 attempt gets 15 s to its response headers after the gateway lookup (OmniRoute's own probe,
 up to two 4 s rounds when no gateway is cached) and is then abandoned for MLX, which serves
@@ -1634,7 +1984,7 @@ consecutive transport or startup failures outside a cooldown (a failure more tha
 the last counted one starts the count over), a relay alias cools down for 30 s, doubling up to
 5 min. A busy MLX slot and timeouts cut short by an execution budget never count; any HTTP
 answer, a success or an error status, proves the transport works and ends the streak and the
-cooldown. `ahub status` shows `cooling down until ...` on the backend line and `events.jsonl`
+cooldown. `ahub status` shows a `cooling` detail with a relative time and failure count and `events.jsonl`
 records `cooldown` events.
 Progress judgements suggest reassignment; they never change task ownership.
 
@@ -1774,7 +2124,8 @@ sequential seeded-guard job. Main and release jobs reuse only a successful full
 PR or push check with the identical Git tree and all three successful jobs. An absent or
 unreadable result runs the main gate again and refuses release. A manual
 `prepare_bundle` dispatch builds reviewable plugin assets without publishing;
-it is never accepted as full-gate evidence.
+it is never accepted as full-gate evidence. A `timing_runs=20` dispatch runs only the daemon, PII screen and ACP
+test files twenty times on macOS, in its own concurrency group; it does not replace the full gate.
 
 ## Preview initialization and native launch
 
@@ -1803,9 +2154,19 @@ ownership, bind servers or start agents, sidecars or models.
 `ahub status`, `ahub tail` and the dashboard show native context occupancy,
 source and measurement freshness. Claude readings come from the status-line
 tee installed by `ahub claude`; Codex readings come from its current thread's
-native token-usage updates. Pi and other unsupported surfaces show unknown.
+native token-usage updates. Kimi readings come from its ACP `usage_update`
+(`{used, size}`, source `acp_usage_update`): context occupancy, not
+consumption, so no token total is counted from it; the adapter accepts the
+update whenever it names the current session, because 2.1.1's source text
+emits it after the turn settles (read from the source, not yet observed live
+through the hub; docs/smoke.md), so the peer may already be idle.
+Pi and other unsupported surfaces show unknown.
 A stale or disconnected reading is unknown, not 0%. Codex's accumulated session
-usage is never used as context occupancy.
+usage is never used as context occupancy. In the tail (`ahub tail` and the
+console's stream) a reading prints once and repeats only when the line changes
+apart from its time: the printed percentage, the freshness, the source, or known
+against unknown. Status, the Peers panel and the dashboard always read the
+latest reading with its time.
 
 Context-triggered checkpoints are disabled by default. To enable them, add
 this to `.agenthub/config.json` and restart the daemon deliberately:
@@ -1817,8 +2178,9 @@ this to `.agenthub/config.json` and restart the daemon deliberately:
 `gate` is a fraction between 0 and 1; 0 disables checkpoint requests.
 `stale_min` must be positive. A reading at or above the threshold records a metadata-only
 event and console notice, then asks an attached Claude/Codex with active work
-for a checkpoint when no checkpoint request is already outstanding. The request
-supplies a `request_id`; include that id with
+for a checkpoint when no checkpoint request is already outstanding (an ACP peer
+such as Kimi gets the event and notice only; no checkpoint request is sent to it
+yet). The request supplies a `request_id`; include that id with
 `hub_checkpoint {summary, request_id}`. Repeated high readings do not repeat
 it until a fresh below-threshold reading or new session rearms the crossing.
 
@@ -1849,13 +2211,88 @@ Pi exit records show code or signal, turn/tool activity, whether teardown was
 expected, startup status, and the last tool's name without arguments. An
 unobserved native-terminal exit records unknown OS status. The console notice
 gives the next action, usually `ahub pi`, or says why automatic start is held.
-A requested replacement says the new owner is starting; internal failure
-teardown directs session inspection rather than claiming a person requested it.
+A replacement says the owner stopped for a new Pi owner, including when the
+replacement is refused. Internal failure teardown directs session inspection
+rather than claiming a person requested it. Replacement, failed-start and
+active-turn guidance appears before automatic-start-disabled guidance.
 
 With `pi.auto_start`, an unexpectedly idle headless Pi is resumed on its
 verified persisted session once. The 60-second retry window rearms when the
 attempt settles; a second exit inside that window stays offline. Missing or
 invalid session history has no fresh fallback; inspect it before running
 `ahub pi`. Active turns/tools, failed startup, a superseded owner, native
-terminal exits, requested shutdown and recovery operations suppress this
-restart. Crash recovery's #66 recorded-session/fresh-start choices are unchanged.
+terminal exits, a requested stop (`ahub stop pi`), requested shutdown and
+recovery operations suppress this restart. Crash recovery's #66
+recorded-session/fresh-start choices are unchanged.
+
+### Pi/local unanswered approvals
+
+An expired Pi/local request says "approval expired: no person answered". The
+worker must not retry that call; hand the task off or stop. Two consecutive
+unanswered requests end the current turn and produce a console notice. An
+expiry counts only when no person answered a request of that peer while it
+was pending, so a batch of parallel requests does not stop a worker whose
+person is answering; a grant served from the always-allow cache or a permission
+mode is not an answer and does not reset the count. A new turn also resets it. The turn ends
+through the same abort path an execution-budget stop uses: Pi stays attached
+and the sender is told the approval reason. Supported Pi and local settle that
+delivery after reporting the reason, so the next queued message runs without
+an operator release. Pi clears turn-only approval state at settlement; the
+person's idle `!command` remains usable and a later failure keeps its own reason.
+A Pi whose extension cannot abort
+a turn (an older hub extension or a runtime without native abort support) goes
+offline instead. Inspect prior work, restart with `ahub pi`, find the held
+delivery with `ahub queue list --peer pi`, inspect it with `ahub queue show <id>`,
+then explicitly settle it with `ahub queue resolve <id> --action completed|retry|discard --reason "<text>"`.
+A cancelled tool withdraws
+its request immediately; a later `ahub permit` answer is refused and cannot
+create an always-allow grant. Tool requests from a previous turn or an older
+extension missing session/turn identity are refused before execution; restart
+Pi with the current hub extension rather than retrying that old call.
+
+The dashboard shows the number of waiting Pi/local requests and where to
+answer them. It remains deny-only for those private rows. Use the console or
+`ahub tail` to inspect the request, then `ahub permit` to answer it. Recovery
+stops waiting for an approval when that request is withdrawn.
+
+A person's `!command` in the managed Pi TUI runs through the managed sandbox
+without asking for another hub approval. Its reported exit code comes from
+the process, including a nonzero exit; text printed by the command does not
+set its status. This exception requires the current idle TUI owner's reserved
+request and does not apply to model-origin bash calls.
+
+### Stop one headless peer
+
+`ahub stop kimi`, `ahub stop pi` (headless mode), or `ahub stop local` stops
+that hub-owned runtime through its adapter and prints `<peer>: offline`.
+The hub and its other peers keep running. The command is human-only;
+agent shells and tools-role connections cannot use the `peer_stop` request.
+An older hub that does not know the request needs an upgrade before this
+command is available.
+
+Claude, Codex and Pi in TUI mode belong to their native terminals: end them
+in those terminals. The upgrade plan's end-agents action closes a TUI through
+its verified terminal binding and uses `peer_stop` for a headless runtime.
+Unknown and already-offline peers are refused with a next action. A recovery
+operation or shutdown holds stop mutations; finish that operation first.
+A peer's concurrent start, stop or permission-mode change must also finish
+before another lifecycle change is admitted, and an upgrade or restart does not
+prepare the hub while a stop is in progress.
+
+A requested stop does not automatically restart a peer. A person or an authorized
+conductor can explicitly start it again; conductor starts remain permitted. Start it with
+`ahub kimi`, `ahub pi` or `ahub local`. Pending approvals are withdrawn before
+teardown. An in-flight delivery becomes `needs_review`, with a reason naming
+the requested stop; it is neither completed nor automatically replayed.
+Inspect partial effects with `ahub queue list --peer <peer>` and
+`ahub queue show <delivery-id>`, then explicitly choose `completed`, `retry`
+or `discard` with `ahub queue resolve`. Restarting alone does not release
+that held delivery. Work queued behind an idle stopped peer is retained and
+runs when it is started again, subject to its existing pauses and holds. New work
+arriving during teardown stays queued too, even if a console resume runs meanwhile.
+An idle Pi with no message/file yet records its live empty-session proof before
+stopping, so its next start resumes that identity. A Pi stopped after its first
+prompt was accepted but before history was persisted is not treated as empty:
+its delivery is held, and another start refuses until resume history is verified.
+Inspect and settle the held delivery before deliberately replacing that unpersisted
+session by restarting the hub; the stop never silently chooses a fresh session.