@staix/agent-hub 0.12.19 → 0.12.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +3 -3
- package/docs/agent-notes/adapters.md +1 -0
- package/docs/agent-notes/bus.md +1 -0
- package/docs/agent-notes/daemon.md +2 -1
- package/docs/agent-notes/models.md +3 -0
- package/docs/agent-notes/tasks.md +3 -2
- package/docs/events.md +4 -1
- package/docs/operations.md +413 -17
- package/docs/quickstart.md +1 -1
- package/docs/security.md +17 -10
- package/docs/smoke.md +58 -0
- package/docs/specs/2026-09-19-agent-hub-design.md +399 -3
- package/docs/specs/2026-09-20-pi-local-models-design.md +2 -0
- package/docs/specs/2026-09-20-upgrade-recovery-design.md +287 -0
- package/docs/specs/2026-10-04-switchyard-source-port-design.md +10 -0
- package/docs/verified.json +13 -13
- package/package.json +1 -1
- package/plugins/agent-hub/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-hub/server.js +63 -25
- package/src/adapters/claude-channel.ts +61 -20
- package/src/adapters/codex-appserver.ts +3 -0
- package/src/adapters/local-worker.ts +4 -0
- package/src/cli/console-state.ts +325 -102
- package/src/cli/console.ts +44 -23
- package/src/cli/help.ts +120 -0
- package/src/cli/identity.ts +3 -3
- package/src/cli/launch.ts +2 -0
- package/src/cli/main.ts +171 -75
- package/src/cli/preview.ts +1 -0
- package/src/cli/reset.ts +194 -0
- package/src/cli/status-lines.ts +9 -5
- package/src/cli/terminal-recovery.ts +140 -36
- package/src/cli/upgrade-runtime.ts +274 -56
- package/src/cli/upgrade.ts +366 -41
- package/src/hub/ask.ts +1 -1
- package/src/hub/board.ts +15 -7
- package/src/hub/bus.ts +60 -4
- package/src/hub/child-process.ts +1 -0
- package/src/hub/conductor.ts +38 -6
- package/src/hub/control-client.ts +5 -3
- package/src/hub/daemon.ts +56 -9
- package/src/hub/events.ts +7 -1
- package/src/hub/execution-budget.ts +15 -6
- package/src/hub/hub-tools.ts +4 -4
- package/src/hub/inference.ts +136 -1
- package/src/hub/lifecycle.ts +7 -1
- package/src/hub/peers.ts +2 -0
- package/src/hub/recovery-store.ts +51 -3
- package/src/hub/report.ts +43 -1
- package/src/hub/restart.ts +25 -0
- package/src/hub/routing.ts +48 -16
- package/src/hub/snapshots.ts +3 -2
- package/src/hub/tasks.ts +236 -83
- package/src/local/deny.ts +2 -1
- package/src/models/mlx.ts +11 -8
- package/src/models/ollama.ts +2 -2
- package/src/models/relay.ts +190 -42
- package/src/models/route/relay-selector.ts +43 -31
- package/src/models/route/runtime.ts +17 -7
- package/src/models/route/signals.ts +12 -1
- package/src/models/route/stage.ts +57 -3
- package/src/omniroute/client.ts +1 -1
- package/src/ui/index.html +1 -0
- package/templates/AGENTS.block.md +1 -1
- package/templates/routing.toml +8 -0
package/docs/operations.md
CHANGED
|
@@ -1,8 +1,25 @@
|
|
|
1
1
|
# Operations guide
|
|
2
2
|
|
|
3
|
-
This guide describes ahub 0.12.
|
|
3
|
+
This guide describes ahub 0.12.20 and control protocol 16. Live verification
|
|
4
4
|
results and remaining prerequisites are recorded separately in [the smoke ledger](smoke.md).
|
|
5
5
|
|
|
6
|
+
## Command help
|
|
7
|
+
|
|
8
|
+
`ahub help`, `ahub --help` and `ahub -h` print the commands grouped by section;
|
|
9
|
+
`ahub help <command>` prints only that command's entries, and so does
|
|
10
|
+
`ahub <command> -h` (or `--help`) when the flag is the only argument; anywhere else
|
|
11
|
+
it is an ordinary argument, and `ahub claude` and `ahub codex` pass it on to the
|
|
12
|
+
agent. An unknown command
|
|
13
|
+
prints `unknown command "<name>"; run ahub help` and exits non-zero. Usages start
|
|
14
|
+
at two spaces and descriptions at one fixed column, wrapped at word boundaries to
|
|
15
|
+
the terminal width, read as at least 80 and at most 100 columns.
|
|
16
|
+
|
|
17
|
+
Help uses the console palette: section headings bold cyan, `ahub <command>` words
|
|
18
|
+
cyan, arguments and descriptions the terminal's default foreground. It has no
|
|
19
|
+
`--color` flag: color is on only when standard output is a terminal, `TERM` is
|
|
20
|
+
not `dumb` and `NO_COLOR` is empty or absent. Pipes and redirects get plain text
|
|
21
|
+
without escape sequences, laid out for 80 columns.
|
|
22
|
+
|
|
6
23
|
## Operator console and panels
|
|
7
24
|
|
|
8
25
|
`ahub console` combines the existing tail stream with peer, quota, approval-age
|
|
@@ -23,11 +40,53 @@ asking, under the same path guard and sandbox; a Pi restart or replacement, or a
|
|
|
23
40
|
hub restart, drops it. The hub log records each granted call by tool name only.
|
|
24
41
|
The dashboard can only deny Pi's requests.
|
|
25
42
|
|
|
26
|
-
Tab toggles stream and panels; `ahub console --panels` starts in panels.
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
43
|
+
Tab toggles stream and panels; `ahub console --panels` starts in panels. A panel
|
|
44
|
+
screen has a header (the project's directory name, then the tabs), the panel and
|
|
45
|
+
a three-line footer, separated by ASCII `-` rules; the stream has one rule above
|
|
46
|
+
its footer. The active tab is bracketed, as in `[2 Approvals]`, so it reads
|
|
47
|
+
without color. The Approvals tab counts pending requests and the Queue tab
|
|
48
|
+
counts `needs_review` deliveries, in the attention and failure tones.
|
|
49
|
+
|
|
50
|
+
Peers, Approvals, Tasks and Queue are tables with a header row. A column is as
|
|
51
|
+
wide as its widest cell, up to a third of the terminal; the last column takes
|
|
52
|
+
the rest and is cut with `...`.
|
|
53
|
+
|
|
54
|
+
| Panel | Columns |
|
|
55
|
+
|---|---|
|
|
56
|
+
| Peers | PEER STATE LINK Q ! REVIEW PAUSE QUOTA MODEL |
|
|
57
|
+
| Approvals | ID PEER LEFT TITLE |
|
|
58
|
+
| Tasks | ID STATE OWNER REVIEWER CLASS AGE TITLE |
|
|
59
|
+
| Queue | ID PEER STATE REV AGE |
|
|
60
|
+
|
|
61
|
+
Zero counters and unknown values read `-`. Durations read `45s`, `12m`, `3h05m`
|
|
62
|
+
or `2d03h`; quota resets and budget pauses read `in 2h13m`, and a pause by the
|
|
63
|
+
console user reads `user`. When a peer's status carries #205's optional
|
|
64
|
+
`toolsOnly` field (a Claude session attached without channel pushes), LINK reads
|
|
65
|
+
`tools-only`. Below the Approvals table, a rule and the selected
|
|
66
|
+
request: its title, its allow options numbered as `a` offers them, and `d deny`.
|
|
67
|
+
A title too long for the panel is marked `(more)` in the label column; Enter
|
|
68
|
+
shows it whole. `d deny` always shows; options that do not fit are counted on a
|
|
69
|
+
`(more)` line. The first request is selected when it arrives; `[` and `]` (or
|
|
70
|
+
j/k in the panel) select another. `a`, `d` and `v` act on the selected request
|
|
71
|
+
only: when it closes, the selection clears with a notice instead of moving to
|
|
72
|
+
another request, and a request that arrives later waits for `[` or `]`. Moving
|
|
73
|
+
the selection cancels a pending allow choice or confirmation; the choice prompt
|
|
74
|
+
names its request. Agent-written text below a field, and option names in
|
|
75
|
+
prompts, are quoted unless they are a single word starting with a letter, so
|
|
76
|
+
punctuation, a newline or a bare number in them cannot imitate the console's own
|
|
77
|
+
structure. Under the `?` key table only `?`, Escape and `q` act.
|
|
78
|
+
|
|
79
|
+
Arrow keys or j/k move, Enter opens a detail of labeled fields (relative times,
|
|
80
|
+
never JSON), Escape returns, and `?` shows the keys grouped by panel; in the
|
|
81
|
+
stream, `?` prints that table and `v` prints the selected request as its push
|
|
82
|
+
showed it. The footer's last line lists only the keys that act in the current
|
|
83
|
+
mode, panel and state. A notice shows on the footer's second line until the next
|
|
84
|
+
key or for about ten seconds: errors and refusals (a refused command, a request
|
|
85
|
+
error, the size refusal) in the failure tone, notices that only inform (a closed
|
|
86
|
+
or missing selection) in the attention tone. With a detail open, `p`, `r` and `a`
|
|
87
|
+
act on the item the detail shows. `:` enters a command; its output lines
|
|
88
|
+
start at column 4, as message bodies do. Assignment, delivery resolution and
|
|
89
|
+
allow decisions require confirmation; delivery resolution requires a reason. Tasks use the same public redaction as the board. Panels need at least
|
|
31
90
|
80 columns by 24 rows; smaller terminals stay in stream mode. Task and queue
|
|
32
91
|
polling runs only while the corresponding panel is visible. Leaving restores
|
|
33
92
|
the terminal and returning from panels replays the bounded stream buffer.
|
|
@@ -45,7 +104,7 @@ management or the final reset. Invalid values fail before connecting.
|
|
|
45
104
|
| Success and availability | Green | Idle peer, approved task |
|
|
46
105
|
| Waiting and attention | Yellow | Busy/paused peer, pending approval, confirmation, review/ready task, important priority |
|
|
47
106
|
| Failure and intervention | Red | Failed/check-failed task, undeliverable/overflow, `needs_review` queue, denial request or expired/cancelled approval |
|
|
48
|
-
| Metadata | Bright black | Ages
|
|
107
|
+
| Metadata | Bright black | Ages, remaining approval time and rules |
|
|
49
108
|
|
|
50
109
|
Offline peers, primary titles, action details and message body lines keep the
|
|
51
110
|
terminal's default foreground. Labels and prompts remain readable without
|
|
@@ -305,7 +364,10 @@ the routing configuration.
|
|
|
305
364
|
A review request carries a checklist: map the changed signatures and call sites
|
|
306
365
|
to the task's plan (without one, to its detail, which the request then
|
|
307
366
|
includes), read the check result, and list what is
|
|
308
|
-
unmet (`hub_review` takes `unmet`, `ahub review ... --unmet <item>`).
|
|
367
|
+
unmet (`hub_review` takes `unmet`, `ahub review ... --unmet <item>`). A reviewer
|
|
368
|
+
that missed the request, and the owner, read the done summary and check result
|
|
369
|
+
with `hub_task_show {id}`: the task's public view with its history, as the
|
|
370
|
+
conductor sees it, and only a stub for a PII task. Other peers are refused. The hub
|
|
309
371
|
records how each review turned out, per implementer, reviewer and class:
|
|
310
372
|
approved; caught (changes were requested and the owner's redo was approved);
|
|
311
373
|
contradicted (within a week, work on the same file or symbol failed its check
|
|
@@ -327,7 +389,8 @@ one line with its next message, the plan included, at no turn of its own. When a
|
|
|
327
389
|
task is done (after its check passes, when one is configured), the owners of
|
|
328
390
|
open tasks on the same paths or symbols get a message with the changed files,
|
|
329
391
|
the plan's signatures and the first line of the summary, each left out when it
|
|
330
|
-
matches a PII pattern
|
|
392
|
+
matches a PII pattern (the summary also when `signals.pii_screen = "local"` did
|
|
393
|
+
not clear it); nobody else does. PII
|
|
331
394
|
tasks are left out on both sides.
|
|
332
395
|
|
|
333
396
|
A task can wait for others: `hub_task_propose` takes `after: [ids]` (`ahub task
|
|
@@ -340,8 +403,36 @@ to the next run.
|
|
|
340
403
|
`ahub board --ready` and
|
|
341
404
|
`hub_task_list {ready: true}` list the proposed tasks with nothing left to wait
|
|
342
405
|
for. Dependencies are fixed when a task is proposed and can only name tasks that
|
|
343
|
-
already exist, so they cannot form a cycle.
|
|
344
|
-
|
|
406
|
+
already exist, so they cannot form a cycle.
|
|
407
|
+
|
|
408
|
+
An owner named for a task that waits (`owner` with `after`, `--owner` with
|
|
409
|
+
`--after`) is its reserved owner: the task still waits, and when it is ready
|
|
410
|
+
routing offers it to that peer first, not to the first idle peer of the class.
|
|
411
|
+
A reserved owner that is offline, paused, failing, not attached or excluded is
|
|
412
|
+
passed over: routing proceeds as usual, the console and hub.log say which
|
|
413
|
+
reservation was passed over and why, and the assignment's history note keeps
|
|
414
|
+
it, unless it is no news (the reserved peer is the owner the task moves away
|
|
415
|
+
from, or it already refused the task). A peer the task was declined for (by
|
|
416
|
+
itself or by the console) or escalated away from after repeated
|
|
417
|
+
changes_requested or by hand stays excluded on every later reroute; an owner
|
|
418
|
+
released as gone, or moved by the hub after a failed delivery or inference, is
|
|
419
|
+
excluded only from that move and may get the task again from its reservation
|
|
420
|
+
later. Your own `ahub task assign` is not blocked by those exclusions, and it
|
|
421
|
+
drops the agent's reservation. The PII constraint and capability limits
|
|
422
|
+
still apply, so a PII task goes to `local` or nobody whoever was reserved.
|
|
423
|
+
`ahub task assign` on a task that waits changes its reserved owner instead of
|
|
424
|
+
handing it over. `ahub route explain <id>` names the reserved owner; once the
|
|
425
|
+
task is ready it also says whether routing would honor the reservation now, and
|
|
426
|
+
why not.
|
|
427
|
+
|
|
428
|
+
The peer that proposed a task may redirect it with `hub_task_assign` while it
|
|
429
|
+
is `proposed` and nobody ever accepted it, without the conductor role (handing
|
|
430
|
+
it to another peer needs `assign` when the proposer has a capabilities list).
|
|
431
|
+
Once it was accepted, even if a decline or release later put it back in
|
|
432
|
+
`proposed`, or after a person assigned or reserved it from the console (also
|
|
433
|
+
once the hub carried that out), only the conductor and the console move it.
|
|
434
|
+
|
|
435
|
+
An owner offline longer than `tasks.release_after_min`
|
|
345
436
|
(default 30, `0` turns it off) in `.agenthub/config.json` loses its open tasks
|
|
346
437
|
to a peer routing can give them to; with nobody to take them they stay, and a
|
|
347
438
|
paused peer or a hub in a recovery operation is left alone.
|
|
@@ -393,6 +484,13 @@ ahub report --by task # tokens, turns and wall time per task and class
|
|
|
393
484
|
ahub export --since 24h # the raw events as JSON lines, for your own analysis
|
|
394
485
|
```
|
|
395
486
|
|
|
487
|
+
Per peer and model route, `ahub report` also counts model changes between consecutive
|
|
488
|
+
decisions, how many of them landed inside tool loops and, with `stay_switch` on, sessions
|
|
489
|
+
and the planner's switches (#197). With `stay_switch = "off"` session boundaries are not
|
|
490
|
+
recorded; `shadow` routes exactly as `off` does, so it is the baseline to compare `enforce` with.
|
|
491
|
+
A route event records the decision, so a load-moved request that its MLX fallback served counts
|
|
492
|
+
as a change to `dgx/fast`; the relay's request journal records which backend served it.
|
|
493
|
+
|
|
396
494
|
`ahub report` counts the same overlap warnings as `scripts/overlaps.ts`, from the
|
|
397
495
|
structured events instead of log lines. `--by task` uses the task each usage and
|
|
398
496
|
token record was attributed to when it was written: the delivery that started the
|
|
@@ -740,6 +838,30 @@ the system marks it `needs_review`. Establish evidence before choosing
|
|
|
740
838
|
disconnects, cancellation, and partial effects are uncertain; do not replay
|
|
741
839
|
them automatically.
|
|
742
840
|
|
|
841
|
+
### Claude sessions without channel pushes
|
|
842
|
+
|
|
843
|
+
Only `ahub claude` starts Claude Code with the development-channel flag. A
|
|
844
|
+
Claude session started another way (plain `claude`, an IDE) keeps the hub tools
|
|
845
|
+
but cannot show pushes, so it attaches tools-only (protocol 16, issue #205):
|
|
846
|
+
|
|
847
|
+
- Its messages stay queued. `ahub status` shows `queued N` and `tools-only:
|
|
848
|
+
messages wait for hub_inbox; for pushes restart Claude with ahub claude`; the
|
|
849
|
+
console footer shows `tools-only: ahub claude` and the dashboard shows the
|
|
850
|
+
status line.
|
|
851
|
+
- The session reads them with `hub_inbox`, at most ten messages at a time plus
|
|
852
|
+
the hub's recall note when one waits (it says how many still wait). Each read is one `completed` delivery (`read through
|
|
853
|
+
hub_inbox` in `ahub queue list`); nothing is ever `accepted`, and the session
|
|
854
|
+
has no `hub_delivery_done`. A read the model never saw (the tool call was
|
|
855
|
+
cancelled, the plugin died) cannot be retried: `ahub queue show <id>` of the
|
|
856
|
+
`completed` row prints the messages to send again.
|
|
857
|
+
- Whatever holds pushes holds `hub_inbox` too (`needs_review`, recovery,
|
|
858
|
+
`ahub pause`, a budget pause, a conductor hold): it reads nothing and says
|
|
859
|
+
which hold applies.
|
|
860
|
+
- Restarting Claude with `ahub claude` takes the peer over and pushes what still
|
|
861
|
+
waits; the plain session stands by. The other way round, a plain session
|
|
862
|
+
started while an `ahub claude` session holds the peer stands by and attaches
|
|
863
|
+
only after that session leaves.
|
|
864
|
+
|
|
743
865
|
## Graceful shutdown
|
|
744
866
|
|
|
745
867
|
Before stopping, let active turns and approvals settle when possible:
|
|
@@ -764,6 +886,93 @@ so a restarted hub never launches beside a survivor. A hub whose project root
|
|
|
764
886
|
or state directory was deleted stops itself within about 10 seconds; the
|
|
765
887
|
dashboard manager does the same when its home directory vanishes.
|
|
766
888
|
|
|
889
|
+
## Resetting a project's hub
|
|
890
|
+
|
|
891
|
+
`ahub reset` returns a project's hub to a clean state. Without `--yes` it is a
|
|
892
|
+
dry run: it lists what it would settle, clear, archive and keep, by delivery
|
|
893
|
+
id, peer name and count, never task or message text, and changes nothing.
|
|
894
|
+
With `--yes` it first stops a running hub the way `ahub kill` does, then acts.
|
|
895
|
+
|
|
896
|
+
```bash
|
|
897
|
+
ahub reset # dry run, runtime scope
|
|
898
|
+
ahub reset --yes # runtime reset
|
|
899
|
+
ahub reset --all # dry run, full reset
|
|
900
|
+
ahub reset --all --yes # full reset
|
|
901
|
+
```
|
|
902
|
+
|
|
903
|
+
The runtime reset (the default):
|
|
904
|
+
|
|
905
|
+
- discards every `queued` and `needs_review` delivery with reason `reset`
|
|
906
|
+
through the delivery journal, one `resolution_history` entry each. Work
|
|
907
|
+
that was dispatching or accepted when the hub stopped is held as
|
|
908
|
+
`needs_review` first, so it is discarded too;
|
|
909
|
+
- clears manual peer holds (`ahub pause`), budget pauses and conductor holds;
|
|
910
|
+
- drops the agent session resume pointers `sessions.json`,
|
|
911
|
+
`claude-session.json` and `claude-context.json`, so the next launch starts
|
|
912
|
+
new agent sessions. Pi transcripts under `pi-sessions/` stay;
|
|
913
|
+
- removes the manifest (`status.json`, `control-token`, `hub.pid`) a hub left
|
|
914
|
+
when it did not stop cleanly, once no process behind it is alive;
|
|
915
|
+
- keeps the board (tasks, reviews, outcomes, turns, touches), `hub.log`,
|
|
916
|
+
`events.jsonl` and `cli-audit/`, recovery records, execution budgets,
|
|
917
|
+
configuration and files the hub does not own.
|
|
918
|
+
|
|
919
|
+
The full reset (`--all`) moves the whole state directory, unchanged, to
|
|
920
|
+
`.agenthub/archive/state-<UTC time>/` (mode 0700) and starts an empty state
|
|
921
|
+
directory holding only `project.json`, so the project keeps its id and
|
|
922
|
+
registration; the next `ahub up` starts with an empty board and queue.
|
|
923
|
+
`.agenthub/config.json`, `config.local.json` and `routing.toml` live outside
|
|
924
|
+
the state directory and are not touched. The archive keeps the task text,
|
|
925
|
+
PII included, so it stays in the project: the reset writes
|
|
926
|
+
`.agenthub/archive/.gitignore` (`*`) so git ignores it, and the local
|
|
927
|
+
worker's and Pi's denylist (file tools, sandbox, memory capture) covers
|
|
928
|
+
`.agenthub/archive` as it covers `.agenthub/state`. Turn snapshots leave the
|
|
929
|
+
archive out as they leave the state out. Nothing deletes or prunes
|
|
930
|
+
archives; removing one is a manual act.
|
|
931
|
+
|
|
932
|
+
To restore an archive, stop the hub, move the current state directory into
|
|
933
|
+
the archive too (where git and the local worker's denylist cover it), then move
|
|
934
|
+
the archive back:
|
|
935
|
+
|
|
936
|
+
```bash
|
|
937
|
+
ahub kill
|
|
938
|
+
mv .agenthub/state .agenthub/archive/state-$(date -u +%Y%m%dT%H%M%SZ)
|
|
939
|
+
mv .agenthub/archive/state-<UTC time> .agenthub/state
|
|
940
|
+
ahub up
|
|
941
|
+
```
|
|
942
|
+
|
|
943
|
+
Only a person runs `ahub reset`: an agent shell is refused before anything
|
|
944
|
+
happens, as for `kill`, `restart` and `upgrade`. It is refused, with nothing
|
|
945
|
+
changed:
|
|
946
|
+
|
|
947
|
+
- while an upgrade or recovery operation is open (the machine's recovery lock,
|
|
948
|
+
or a running hub that reports an unreleased recovery);
|
|
949
|
+
- when no registration matches this project and state directory. A running
|
|
950
|
+
hub must then be stopped with its matching CLI; a stopped project is
|
|
951
|
+
registered again by `ahub up` (then `ahub reset --yes` stops it first). When
|
|
952
|
+
the project is registered with another state directory, the error names
|
|
953
|
+
`ahub --project <id> reset` instead;
|
|
954
|
+
- when the hub's ownership cannot be verified, as `ahub kill` refuses it;
|
|
955
|
+
- for `--all`, when the state directory is not `<root>/.agenthub/state`
|
|
956
|
+
(`AGENTHUB_STATE_DIR` or a symlink elsewhere): archive such a directory by
|
|
957
|
+
hand after `ahub kill`;
|
|
958
|
+
- for `--all`, when `.agenthub/archive` is a symlink or not your own
|
|
959
|
+
directory, or its `.gitignore` is not a regular file: anything with write
|
|
960
|
+
access to the project could otherwise send `hub.db` out of it. Inspect and
|
|
961
|
+
remove what is there, then run the reset again.
|
|
962
|
+
|
|
963
|
+
After stopping the hub the reset holds the project's registry claim, the one a
|
|
964
|
+
daemon takes to run, until it is done, so no hub starts under it; a hub that
|
|
965
|
+
started in between makes it stop with nothing reset; a crashed hub's leftover
|
|
966
|
+
manifest does not count as one. If a step fails after the stop, the error says
|
|
967
|
+
how far it got: rerun `ahub reset --yes` to finish a runtime reset (every step
|
|
968
|
+
can run again), unless the error says the state cannot be read (a damaged
|
|
969
|
+
`hub.db` or journal fails the same way every time): `ahub reset --all --yes`
|
|
970
|
+
archives such a state directory as it is. A full reset names the archive once
|
|
971
|
+
the state directory has moved.
|
|
972
|
+
It never touches claude-mem: notes saved with `hub_remember` are shared memory,
|
|
973
|
+
not hub state. Claude Code sessions attached to the hub lose their hub session;
|
|
974
|
+
relaunch them with `ahub claude`.
|
|
975
|
+
|
|
767
976
|
## Orphaned daemons
|
|
768
977
|
|
|
769
978
|
Daemons from before those guards, or daemons whose project directory was
|
|
@@ -792,21 +1001,21 @@ Rows without a live process are stale registrations; forget them with
|
|
|
792
1001
|
|
|
793
1002
|
Upgrade running projects with the target release's own coordinator. It accepts
|
|
794
1003
|
a running source on control protocol 9 (0.6.x), 10 (0.7.0 through 0.12.0),
|
|
795
|
-
11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15) 14 (0.12.16)
|
|
1004
|
+
11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20), and only
|
|
796
1005
|
a target on its own protocol, so the target's coordinator fits every supported
|
|
797
1006
|
source and carries every recovery fix released up to it. Protocol 8 and older
|
|
798
1007
|
(0.5.x and earlier) are refused as `manual-bootstrap-required`. Run from the
|
|
799
1008
|
project directory, without replacing the global CLI first:
|
|
800
1009
|
|
|
801
1010
|
```bash
|
|
802
|
-
bunx --package @staix/agent-hub@0.12.
|
|
803
|
-
bunx --package @staix/agent-hub@0.12.
|
|
1011
|
+
bunx --package @staix/agent-hub@0.12.20 ahub upgrade --to 0.12.20 --dry-run
|
|
1012
|
+
bunx --package @staix/agent-hub@0.12.20 ahub upgrade --to 0.12.20 --yes
|
|
804
1013
|
```
|
|
805
1014
|
|
|
806
1015
|
| Running now | Coordinator to use |
|
|
807
1016
|
| --- | --- |
|
|
808
1017
|
| 0.6.x (protocol 9) | the target's, through `bunx` as above |
|
|
809
|
-
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15) | the target's, through `bunx` as above |
|
|
1018
|
+
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 (protocol 16) | the target's, through `bunx` as above |
|
|
810
1019
|
| any supported source, with the installed CLI already at the target | `ahub upgrade` below, which is the same coordinator |
|
|
811
1020
|
| 0.5.x or earlier (protocol 8 and older) | not supported: bootstrap by hand with the matching CLI |
|
|
812
1021
|
|
|
@@ -838,17 +1047,19 @@ projects first:
|
|
|
838
1047
|
|
|
839
1048
|
```bash
|
|
840
1049
|
ahub restart --dry-run
|
|
841
|
-
ahub upgrade --to 0.12.
|
|
1050
|
+
ahub upgrade --to 0.12.20 --dry-run
|
|
842
1051
|
```
|
|
843
1052
|
|
|
844
1053
|
Apply only after reviewing the plan:
|
|
845
1054
|
|
|
846
1055
|
```bash
|
|
847
1056
|
ahub restart --yes
|
|
848
|
-
ahub upgrade --to 0.12.
|
|
1057
|
+
ahub upgrade --to 0.12.20 --yes
|
|
849
1058
|
ahub recovery status <operation-id>
|
|
850
1059
|
ahub recovery resume <operation-id>
|
|
851
1060
|
ahub recovery abort <operation-id>
|
|
1061
|
+
ahub recovery dispose <operation-id> --fresh-session <peer> --reason <text>
|
|
1062
|
+
ahub recovery dispose <operation-id> --stop-and-archive --reason <text>
|
|
852
1063
|
```
|
|
853
1064
|
|
|
854
1065
|
The coordinator commits only once the source is quiet: no turn running, no
|
|
@@ -904,6 +1115,20 @@ Kimi and local sessions may start fresh with preserved routing and task context.
|
|
|
904
1115
|
A Claude session that never persisted a transcript (zero turns) also starts fresh:
|
|
905
1116
|
there is nothing to resume, so the upgrade accepts the new session id; a session
|
|
906
1117
|
with a transcript must come back with its original id.
|
|
1118
|
+
A Claude session that `ahub claude` did not launch in a recorded Orca terminal
|
|
1119
|
+
(a plain `claude`, or one whose launcher has ended) is planned as
|
|
1120
|
+
reconnect-only: the plan lists it under `reconnectOnly`, and the command prints
|
|
1121
|
+
`<project>: claude is reconnect-only` on stderr. Nothing in its terminal is
|
|
1122
|
+
closed or relaunched, a `claude-session.json` left by an earlier launch is never
|
|
1123
|
+
used as its target, and the plan does not show that record's id. Its plugin
|
|
1124
|
+
reconnects to the new hub by itself (the coordinator waits up to 90 seconds) and
|
|
1125
|
+
keeps the plugin version it started with until that Claude session restarts.
|
|
1126
|
+
The target release must read recovery waivers; staging refuses one that does
|
|
1127
|
+
not, before any runtime changes. Across a control-protocol change it cannot
|
|
1128
|
+
reconnect; the plan then names a
|
|
1129
|
+
blocker: end the session, or relaunch it with `ahub claude` in Orca, and make a
|
|
1130
|
+
new plan. A refused `--yes`, and every dry-run, prints each blocker on its own
|
|
1131
|
+
`ahub: blocker:` line before the final one.
|
|
907
1132
|
Previously stopped projects stay stopped; only the reviewed running projects
|
|
908
1133
|
are upgraded.
|
|
909
1134
|
|
|
@@ -911,6 +1136,140 @@ Do not run an upgrade with an incompatible active protocol, an unverified
|
|
|
911
1136
|
terminal binding, or an unresolved operation lock. Dry-run performs no package,
|
|
912
1137
|
plugin, daemon, or terminal mutation.
|
|
913
1138
|
|
|
1139
|
+
### A partial operation
|
|
1140
|
+
|
|
1141
|
+
`ahub recovery status <operation-id>` reads the receipt without changing it:
|
|
1142
|
+
its phase, step and error; `runner`, which says whether a runner process holds
|
|
1143
|
+
the operation now (`running` with its pid, or `none`; `stale` marks a receipt
|
|
1144
|
+
that says running with no runner behind it); each project's phase and effect
|
|
1145
|
+
receipts (`closed:<peer>`, `restored:<peer>` as `done`, `pending` or `failed`);
|
|
1146
|
+
whether the shared plugin and the global CLI were installed; and `next`, the
|
|
1147
|
+
commands that apply now, read from every source the operation has not stopped
|
|
1148
|
+
yet (every error of `resume` ends with the same list). `resume` is left out
|
|
1149
|
+
when it can never get past: a source replaced by another instance, or a
|
|
1150
|
+
prepared source that stopped before any commit request. It shows no task or
|
|
1151
|
+
message text. `resume` does not
|
|
1152
|
+
start a second runner while one is alive. Every `next` entry and error names the
|
|
1153
|
+
operation's own coordinator, `bun <preserved source>/src/cli/main.js recovery
|
|
1154
|
+
...`: in the middle of an upgrade the global `ahub` may still be the older
|
|
1155
|
+
release, whose `recovery` lacks these commands. Copy that command line. When
|
|
1156
|
+
an older coordinator started the operation, `dispose` is named from the running
|
|
1157
|
+
release, which has it. What
|
|
1158
|
+
each receipt allows, given what is live, is tabulated in the recovery spec
|
|
1159
|
+
(`docs/specs/2026-09-20-upgrade-recovery-design.md`, "Receipts, evidence and
|
|
1160
|
+
next actions").
|
|
1161
|
+
|
|
1162
|
+
- The source hold lapses after 10 minutes. A later `resume` prepares the same
|
|
1163
|
+
source again and checks its peers again, keeping the receipts, so no terminal
|
|
1164
|
+
is closed twice. It refuses, naming the next step, when the source daemon was
|
|
1165
|
+
replaced, when another operation holds it, or when a peer changed: a peer whose
|
|
1166
|
+
terminal the operation closed must stay closed, a session that joined since
|
|
1167
|
+
must end, every other one must keep its conversation. With nothing closed yet,
|
|
1168
|
+
`abort` cancels the operation (also once its hold has lapsed, or when another
|
|
1169
|
+
daemon or operation holds the source, which it leaves alone) so a new plan
|
|
1170
|
+
can be made, also after a source crashed before any commit was requested
|
|
1171
|
+
(`resume` then stops, as there is nothing to start from, and names abort, or
|
|
1172
|
+
stop-and-archive once anything was done); it
|
|
1173
|
+
refuses while a prepared source cannot be read (its hold may still stand),
|
|
1174
|
+
once a commit request may have been sent (or, for an operation from
|
|
1175
|
+
an older coordinator, which records no such thing, while a prepared source is
|
|
1176
|
+
not running), and once any project has effects the way out is
|
|
1177
|
+
stop-and-archive. `next` offers abort only where abort would succeed, and
|
|
1178
|
+
`resume` (and `--fresh-session`, which resume carries out) only where it can
|
|
1179
|
+
get past what is live: not past a replaced or missing source, a source that
|
|
1180
|
+
crashed before any commit request, a refusal at staging (the target release
|
|
1181
|
+
or the preserved source can never change back), a hub of another control
|
|
1182
|
+
protocol, a target that stopped after it started, or a stopped hub without
|
|
1183
|
+
the operation's restart snapshot.
|
|
1184
|
+
A second `resume` while its own hold still stands checks the peers again
|
|
1185
|
+
before it closes anything.
|
|
1186
|
+
- When a target hub dies after it started (a crash or a reboot), restarting it
|
|
1187
|
+
from its snapshot is not supported in this release (its peers would have to
|
|
1188
|
+
be relaunched against the new hub): `resume` is not offered and the way out
|
|
1189
|
+
is stop-and-archive.
|
|
1190
|
+
- A Codex conversation comes back only when a rollout file naming its thread
|
|
1191
|
+
exists under `sessions/` of the store the restored terminal uses (the
|
|
1192
|
+
recorded `CODEX_HOME`, else `~/.codex`). Codex writes that file with the
|
|
1193
|
+
first message, so a thread the hub saw Codex start, with no turn on it since,
|
|
1194
|
+
is listed under `freshStart` and restarts as a new session, with nothing to
|
|
1195
|
+
lose; the command says so. A thread with turns, or one whose start the hub's
|
|
1196
|
+
log does not show (it was resumed, or the hub is older), blocks the plan when
|
|
1197
|
+
it has no rollout, and so does a store that cannot be read (unknown, not
|
|
1198
|
+
missing). The coordinator checks again before closing the terminal and before
|
|
1199
|
+
creating the new one. To continue without such a conversation, end that Codex
|
|
1200
|
+
session and close its Orca terminal, then `resume`; it stops at restoring
|
|
1201
|
+
Codex, where `--fresh-session codex` below is the way on.
|
|
1202
|
+
- A restored terminal runs its launch in a login shell, so the coordinator
|
|
1203
|
+
watches the launcher, not the terminal: once `ahub codex`, `ahub claude` or
|
|
1204
|
+
`ahub pi` has recorded itself there and its process is gone, or Orca no longer
|
|
1205
|
+
lists that terminal at all, the restoration failed (found within one 5-second
|
|
1206
|
+
wait). A launcher whose process cannot be read counts as running, never as
|
|
1207
|
+
gone. One that dies before recording itself leaves no record to read, and
|
|
1208
|
+
what happens next depends on whether Orca reports the bare login shell it
|
|
1209
|
+
returned to as TUI-idle, which is not verified. If it does, the coordinator
|
|
1210
|
+
stops at once with "original session restoration needs manual verification"
|
|
1211
|
+
(the terminal cannot be mapped to the session). If it does not, it stops with
|
|
1212
|
+
the same error after the 10-minute wait. Either way the receipt stays
|
|
1213
|
+
`pending`; close that terminal in Orca (or wait, if the session may still
|
|
1214
|
+
attach), then `resume`: with no live launcher and no attached session it
|
|
1215
|
+
records `restored:<peer>` `failed` and lists the failed-restoration choices.
|
|
1216
|
+
A `failed` receipt never counts as restored.
|
|
1217
|
+
`resume` settles a `pending` or `failed` receipt by what is live: the planned
|
|
1218
|
+
session attached is the restoration, a launcher still running is waited for,
|
|
1219
|
+
and no second terminal is opened while one may run. A target hub that cannot
|
|
1220
|
+
be read counts as unknown, never as "nothing attached", and so does a launcher
|
|
1221
|
+
record file that cannot be read: `resume` then stops without changing any
|
|
1222
|
+
receipt; when the record file is what cannot be read, inspect it and move it
|
|
1223
|
+
aside, then `resume`. `resume` is refused while a stop-and-archive is partway. A lifecycle command refused by the lock names
|
|
1224
|
+
the operation's own `status` command, which lists what to do next.
|
|
1225
|
+
- `ahub recovery dispose <operation-id> --fresh-session <peer> --reason <text>`
|
|
1226
|
+
applies to a Codex or Claude peer whose restoration failed (Pi is refused: a
|
|
1227
|
+
restored hub resumes Pi's recorded session). It records the lost session or
|
|
1228
|
+
thread id (status shows it under `lostContinuity`) and resumes: the peer
|
|
1229
|
+
starts without its old conversation, and the rest is verified as usual.
|
|
1230
|
+
- `ahub recovery dispose <operation-id> --stop-and-archive --reason <text>`
|
|
1231
|
+
abandons the operation. It checks every project first and refuses while a
|
|
1232
|
+
hub is unreachable, starting, stopping or of another control protocol; `next`
|
|
1233
|
+
then says `wait until <project>'s hub settles, then ... recovery status
|
|
1234
|
+
<id>` instead of offering it. A hub that never answers (its pid is alive,
|
|
1235
|
+
nothing replies): take the `pid` from that project's
|
|
1236
|
+
`.agenthub/state/status.json`, check with `ps -p <pid> -o command=` that it
|
|
1237
|
+
is that project's hub daemon, end it by hand, then run `status` again. If
|
|
1238
|
+
that pid now belongs to another process (reused after a reboot), move the
|
|
1239
|
+
stale `status.json` aside instead, then run `status` again; likewise when
|
|
1240
|
+
`next` says `wait: runner <pid>` and that pid is not an `ahub recovery-run`
|
|
1241
|
+
process, move the operation's `~/.agenthub/recovery/<id>.json.runner.db`
|
|
1242
|
+
aside, then run `status` again. A
|
|
1243
|
+
project whose directory is
|
|
1244
|
+
gone is only recorded (`ahub doctor --orphans` lists a hub left running
|
|
1245
|
+
there). It records its decision before it acts. It releases a source hold of
|
|
1246
|
+
its own (that hub keeps running), stops a target hub it started and has not
|
|
1247
|
+
released (one an older coordinator started is stopped at that release's
|
|
1248
|
+
control protocol), moves that operation's
|
|
1249
|
+
`restart.json` aside as `restart.abandoned.<hash>.json` and leaves any other
|
|
1250
|
+
hub running; only then is the operation recorded `cancelled` and the lock
|
|
1251
|
+
released. If it fails partway it keeps the lock, records what it did, and
|
|
1252
|
+
`resume` refuses until you run it again. Queued messages are not delivered
|
|
1253
|
+
from the archived file; a 0.7.0 or later hub reloads the queues it kept in
|
|
1254
|
+
`hub.db` when it starts again. Terminals it closed stay closed: start those
|
|
1255
|
+
sessions again by hand. Replacement terminals it opened stay open but have
|
|
1256
|
+
no hub: Claude's plugin reconnects once a hub runs, Codex needs `ahub codex`
|
|
1257
|
+
again.
|
|
1258
|
+
Start a project whose target ran with that release's CLI (`bunx --package
|
|
1259
|
+
@staix/agent-hub@<version> ahub up`), since it may have changed the task
|
|
1260
|
+
database. The global CLI was not promoted.
|
|
1261
|
+
- Both choices are for a person in a terminal; an agent shell is refused, and
|
|
1262
|
+
the reason is kept in the operation's audit. Run each with the command
|
|
1263
|
+
`next` or the error prints: the operation's own coordinator, or the release
|
|
1264
|
+
you are running when that coordinator predates these commands (its `resume`
|
|
1265
|
+
still runs the old runner, and `next` says what that runner cannot do).
|
|
1266
|
+
`--fresh-session` never applies to such an operation: its runner never
|
|
1267
|
+
records a restoration as `failed`.
|
|
1268
|
+
`--fresh-session` is not offered when the target release cannot read recovery
|
|
1269
|
+
waivers, since it could never release a new session; `status` says so in
|
|
1270
|
+
`freshSession`. A runner record that cannot be read shows as `unknown`, and
|
|
1271
|
+
`resume`, `abort` and `dispose` are refused until it can be read.
|
|
1272
|
+
|
|
914
1273
|
### After an unplanned stop
|
|
915
1274
|
|
|
916
1275
|
While it runs, the hub keeps each attached peer's session identity in
|
|
@@ -1037,9 +1396,46 @@ PII calls require a positively confirmed campus gateway immediately before trans
|
|
|
1037
1396
|
PII turns never enter shared history, memory capture, or progress observation.
|
|
1038
1397
|
The `route`, `advisor`, `progress` and `stuck` events contain identifiers and aggregates only.
|
|
1039
1398
|
|
|
1399
|
+
Session-aware stay/switch (#197) covers `hub/auto` and `stage` routes. The top-level
|
|
1400
|
+
`routing.toml` key `stay_switch` is `off`, `shadow` (default: route events record what the
|
|
1401
|
+
planner would do, routing unchanged) or `enforce`. Enforced, a hard override (compaction,
|
|
1402
|
+
critical failure, repeated failure) escalates at once; a tool-result turn keeps the
|
|
1403
|
+
session's tier; other changes wait for the next user turn, and a de-escalation whose
|
|
1404
|
+
conversation is larger than `max_switch_prefill_tokens` (default 32000) stays; a PII route
|
|
1405
|
+
has no such bound, so its size never shows in a decision. A capable
|
|
1406
|
+
hold then lasts at least until the next user turn; `hold_turns` can extend it past that
|
|
1407
|
+
turn. A tier whose backend
|
|
1408
|
+
cannot hold the conversation is never chosen; no route summarizes or trims the history.
|
|
1409
|
+
|
|
1040
1410
|
Pi exposes `hub/auto` for stage routing when available. Fixed `dgx/coding`, `dgx/fast`
|
|
1041
1411
|
and `mlx/fast` aliases still pin the backend. Automatic MLX selection admits the complete
|
|
1042
1412
|
input, tool schemas and requested output within the configured context window.
|
|
1413
|
+
A `hub/auto` request bound for MLX has two candidates, MLX and its `dgx/fast` fallback; load
|
|
1414
|
+
and cooldowns only change their order, never remove one (#199). A fixed alias (`mlx/fast`,
|
|
1415
|
+
`pi_backend = "mlx"`) keeps its backend first as before. `dgx/fast` goes first when MLX is
|
|
1416
|
+
cooling down (route event `source: "cooldown"`). Load moves are opt-in until they are
|
|
1417
|
+
measured: with `[pi] efficient_wait_ms` set (0 to 119999; 0 tries once; read when the hub
|
|
1418
|
+
first starts Pi, like `dgx_max_context_tokens`, so a change takes a hub restart), a request
|
|
1419
|
+
whose MLX slot is still busy after that wait goes to `dgx/fast` first (`source: "load"`), and
|
|
1420
|
+
so does an enforced tool loop pinned to it (`source: "pin"`); without it nothing moves for load. `dgx/fast`
|
|
1421
|
+
never goes first while it is cooling down, or while its own last dispatch failed in any way,
|
|
1422
|
+
an error status or a failed stream included, until it succeeds or 30 s pass (`ahub status`:
|
|
1423
|
+
`last dispatch failed, no load moves until ...`). No load move or pin happens while the request
|
|
1424
|
+
runs under an execution budget, and enforced, a load move happens only at a user turn. A moved
|
|
1425
|
+
attempt gets 15 s to its response headers after the gateway lookup (OmniRoute's own probe,
|
|
1426
|
+
up to two 4 s rounds when no gateway is cached) and is then abandoned for MLX, which serves
|
|
1427
|
+
with the usual slot wait. The bound ends with the headers: a stream that stalls or fails
|
|
1428
|
+
after them is not retried on MLX, and a failed one marks `dgx/fast` failing (a client that
|
|
1429
|
+
disconnects does not). Each backend is tried once per request, so with moves on, a moved attempt
|
|
1430
|
+
cut at 15 s followed by an MLX slot that stays busy past 120 s fails the request (502), where
|
|
1431
|
+
without moves the request would have waited for MLX and then had `dgx/fast` with its full
|
|
1432
|
+
deadline; this is part of what the opt-in accepts until #199 AC5 measures it. After three
|
|
1433
|
+
consecutive transport or startup failures outside a cooldown (a failure more than 10 min after
|
|
1434
|
+
the last counted one starts the count over), a relay alias cools down for 30 s, doubling up to
|
|
1435
|
+
5 min. A busy MLX slot and timeouts cut short by an execution budget never count; any HTTP
|
|
1436
|
+
answer, a success or an error status, proves the transport works and ends the streak and the
|
|
1437
|
+
cooldown. `ahub status` shows `cooling down until ...` on the backend line and `events.jsonl`
|
|
1438
|
+
records `cooldown` events.
|
|
1043
1439
|
Progress judgements suggest reassignment; they never change task ownership.
|
|
1044
1440
|
|
|
1045
1441
|
## Disable local MLX while keeping remote auto routing
|
package/docs/quickstart.md
CHANGED