@staix/agent-hub 0.12.32 → 0.12.33
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +6 -5
- package/docs/agent-notes/bus.md +2 -0
- package/docs/agent-notes/daemon.md +4 -2
- package/docs/agent-notes/tasks.md +1 -1
- package/docs/operations.md +35 -11
- package/docs/quickstart.md +3 -3
- package/docs/security.md +6 -1
- package/docs/specs/2026-09-20-upgrade-recovery-design.md +20 -0
- package/docs/verification/2026-10-18-0.12.33.md +42 -0
- package/docs/verified.json +15 -12
- package/package.json +1 -1
- package/plugins/agent-hub/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-hub/server.js +1 -1
- package/src/cli/reset.ts +13 -1
- package/src/cli/upgrade-presentation.ts +11 -1
- package/src/cli/upgrade-runtime.ts +15 -5
- package/src/cli/upgrade.ts +2 -0
- package/src/hub/bus.ts +133 -14
- package/src/hub/control-client.ts +1 -1
- package/src/hub/daemon.ts +18 -9
- package/src/hub/deferred-hub.ts +60 -0
- package/src/hub/delivery-journal.ts +6 -0
- package/src/hub/restart.ts +12 -1
- package/src/hub/tasks.ts +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
Issue and pull request numbers in the entries for 0.7.7 and earlier refer to the previous repository, archived on 2026-09-30 when this repository's history was rewritten; the one exception is the open smoke-check issue, formerly #12, which moved here as #1. Numbers in newer entries refer to this repository.
|
|
4
4
|
|
|
5
|
+
## 0.12.33
|
|
6
|
+
|
|
7
|
+
Upgrades with cancelled tasks and a managed terminal no longer stop at the lease renewal, and a notice the target sends right after it starts no longer blocks queue verification (#478). The upgrade runs with the target release's own coordinator, so an upgrade from 0.12.32 to this release already uses these fixes. A real upgrade with a managed TUI and cancelled rows (#478) and installed managed-Pi #418 AC8 (#142) remain unverified by this release preparation. Installation and the person's restart require separate receipts.
|
|
8
|
+
|
|
9
|
+
- Preserve target capabilities on recovery lease renewals, defer hub notices until held queue verification finishes, and show recorded blocked errors (#478).
|
|
10
|
+
- Discard deferred notices on reset, retry expired recovery leases after storage failures, and keep abort/archive available for corrupt snapshots (#478).
|
|
11
|
+
|
|
5
12
|
## 0.12.32
|
|
6
13
|
|
|
7
14
|
Settings now reload without a restart where their apply class allows it, and settings changed by an agent never adopt guarded or machine-local values without a person's confirmed reload (#439). The upgrade runs with the target release's own coordinator, so this upgrade already uses the new reattach, continuity and screen changes (#463, #466). The first-party memory pin stays `@staix/agent-mem` 0.3.1; the curated shard migration merged in agent-mem is not released and must not be run before the hub opens memory through shards (#175). Native real-terminal console settings (#439 AC5), a real Orca upgrade with reattach (#463), installed managed-Pi #418 AC8 (#142) and #438 AC7 remain unverified by this release preparation. Installation and the person's restart require separate receipts.
|
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@ Codex, Kimi Code, Pi and a hub-owned local worker (Qwen Code and Jcode optional)
|
|
|
5
5
|
share a message bus and a task board per project and one memory on the machine,
|
|
6
6
|
each attached through its own native interface.
|
|
7
7
|
|
|
8
|
-
Status: 0.12.
|
|
8
|
+
Status: 0.12.33, control protocol 16. Bun + TypeScript, one loopback daemon per
|
|
9
9
|
project. Site: [agent-hub.staix.net](https://agent-hub.staix.net).
|
|
10
10
|
|
|
11
11
|
## What it does today
|
|
@@ -77,15 +77,16 @@ cd <your project> && ahub init && ahub up
|
|
|
77
77
|
|
|
78
78
|
After upgrading from 0.12.30 or earlier, run `ahub jcode --mode headless` once in each
|
|
79
79
|
project that uses Jcode to refresh its owned MCP entry before conductor or dashboard
|
|
80
|
-
starts of Jcode. The entry's format is unchanged in 0.12.
|
|
80
|
+
starts of Jcode. The entry's format is unchanged in 0.12.33; a daemon start that still
|
|
81
81
|
reports the entry missing or stale is fixed by the same command.
|
|
82
82
|
Cancellation needs a daemon advertising `task_cancel`; controlled downgrades
|
|
83
83
|
refuse while cancelled rows exist. For local/Pi implementation, opt in to an
|
|
84
84
|
available `checks.runner = "container"`; issue-backed owners also need authenticated
|
|
85
85
|
spec access. PII work and designated reviewers retain their separate exemptions.
|
|
86
|
-
See [release verification](docs/verification/2026-10-18-0.12.
|
|
87
|
-
still open, including #418 AC8, #438 AC7, the native console settings leg (#439 AC5)
|
|
88
|
-
|
|
86
|
+
See [release verification](docs/verification/2026-10-18-0.12.33.md) for the live legs
|
|
87
|
+
still open, including #418 AC8, #438 AC7, the native console settings leg (#439 AC5),
|
|
88
|
+
a real upgrade with reattach (#463) and a real upgrade with a managed terminal and
|
|
89
|
+
cancelled tasks (#478).
|
|
89
90
|
|
|
90
91
|
Before migrating a memory home with `amem`, upgrade every hub using that home to
|
|
91
92
|
0.12.30 or later, check `ahub memory profile-check`, then quiesce and close all memory
|
package/docs/agent-notes/bus.md
CHANGED
|
@@ -21,6 +21,8 @@ Scope: delivery, digests and condensation, reply addressing, priority and limits
|
|
|
21
21
|
- Observe memory rendering from final `out`, not `originals`, and suppress telemetry when either contains private envelopes. A pull emits rendered/delivered observations only after its journal transaction commits; observer exceptions must never change delivery settlement.
|
|
22
22
|
- Classify automatic held-row completion only through `supersededDeliveryReason`: every original must be a hub presence or a provably moot hub task/review notice. Unknown/live/mixed rows stay held; keep `deliveryTaskContext` display-only. Tag only current-owner notices as owner audience; leave former-owner withdrawals untagged until the task closes. For cancelled tasks, settle earlier workflow notices but retain the stop notice whose `refs.taskCancelled` matches the cancellation revision. Preserve the requested-stop exclusion in `daemon.md`. Record the hub actor through `resolveDelivery`, restoring presence originals through `restorePreface` only for hub completion (person completion stays consumed); run on hold entry, restore, recovery release and board changes, never replay uncertain work. List every hold once in status, console notices and route explanations (#434).
|
|
23
23
|
- `resolveDelivery` mutates queues in place; a refused or failed resolution puts back only that peer's queue and preface and keeps every pause. Never reload a snapshot on a live bus. Keep the selected delivery id and journal revision fixed in console row actions; retry/discard/completed never change a task role or cancel a native turn.
|
|
24
|
+
- Recovery-deferred hub notices (`deferredHub`, #478) are accepted work kept outside the queues and the carried-queue integrity projection: never put them in `queues` or the integrity view while held. Every read of pending work includes them (`pendingEnvelopes`, `withdraw`, `revokeSupervision`, `queuedBacklogIds`, `get`, reset). Past `MAX_DEFERRED_HUB` refuse the publish; never drop a kept notice.
|
|
25
|
+
- Release moves each deferred notice through `publishNow` and checkpoints it before delivery, events or nudges run. A failed checkpoint restores only publication state (the affected queue arrays in place, deferred, seen and withdrawn ids, pending outcomes) and keeps the hold. Never reload a snapshot or touch prefaces, pauses or the recovery mode from it.
|
|
24
26
|
- Do not offer recipient retry to delivery-only local/ACP transports or pass ordinary messages through their hold; their hints direct the person to resolve. Keep current-adapter refusal authoritative and the ephemeral MCP exposure marker advisory; pull-only Kimi TUI retains retry.
|
|
25
27
|
- Route peer retry only through `retryDelivery`: scope explicit-retry history by peer and immutable original ids across all grouped parents, commit monotone consumed-proof inheritance for every grouped non-context original with the checkpoint while keeping resolution audit rows unchanged; exclude hub `presence` originals from every retry proof, warning, inheritance and unknown-marking path, preserve unknown imported lineage fences without disabling fresh work, and admit busy-peer consent only to requeue its own `needs_review` row. Decorate rendered clones after condensation/pull/steer, never originals; reserve the shared journal body cap for the warning and an explicit Unicode-safe cut marker. Keep held hints bounded, body-free and advisory; metadata failure never changes a saved tool result or refusal class (#434).
|
|
26
28
|
- `restore` takes queued work from the journal, never from the committed snapshot once the journal has state (an empty journal, revision 0, is seeded from the commit): from the commit only an empty queue key is carried over, for an offline peer whose key no delivery ever persisted (#234). Never resurrect a non-empty queue from it.
|
|
@@ -20,7 +20,8 @@ Scope: the operator console, control WS and its protocol, state files, the dashb
|
|
|
20
20
|
- Close 4000 is the one close that ends on its own, so it is not in `TERMINAL_CLOSES`: the session stands by and reconnects only once `status.json` reports the peer offline and not `claiming`. One exception (#205, `peerHeld` in `claude-channel.ts`): a session with channel pushes also reconnects when the holder is reported `toolsOnly` and not `claiming`, and takes the peer; the daemon refuses the reverse with 4000. Reconnecting blind would evict whoever holds the id now, and the two would trade it forever. `claim()` happens at hello and `attach()` only after the preface, so the peer reads as offline in between: the daemon writes the status file at the claim and the flag covers that window.
|
|
21
21
|
- Adding a native peer requires testing the command emitted by the actual recovery driver, not only its inspection or launch helpers. Never let a generic non-Codex branch treat a new peer as Claude.
|
|
22
22
|
- Cross-version recovery must use the authenticated source protocol for prepare/commit/abort and the target protocol after startup. Prove the transition against a real prior release before claiming compatibility.
|
|
23
|
-
- Preserve `task_cancel` target-feature preflight before source holds, capture or commit. Reuse stored target features only after verifying the same retained root and digest; legacy receipts still capable of commit must obtain feature proof. Do not apply pre-commit compatibility checks to sources already past commit.
|
|
23
|
+
- Preserve `task_cancel` target-feature preflight before source holds, capture or commit. Reuse stored target features only after verifying the same retained root and digest; legacy receipts still capable of commit must obtain feature proof. Do not apply pre-commit compatibility checks to sources already past commit. Every prepare, including the lease renewal before a terminal close and after a resume, sends the operation's stored `targetFeatures` (#478).
|
|
24
|
+
- Defer hub notices only for an authenticated recovery `prepare` or `commit` whose target features include `recovery_deferred_hub`, or for a restored daemon. Otherwise `setRecoveryHold(true)` moves the deferred notices into queues before commit records expected integrity, and a bare `fenceRecovery` defers nothing. Abort and lease expiry release the bus hold before clearing the operation; if that release fails, keep the operation and let expiry retry on its 1 s timer (#478).
|
|
24
25
|
- Resume a prepared `commitSent` source that is unavailable or database-busy inside the existing bounded shutdown poll; never re-send commit, re-prepare or close terminals again on that path. Require confirmed `stopped` before target startup, refuse a replacement instance, and retain the prepared receipt on timeout.
|
|
25
26
|
- Keep restoration receipts explicitly tagged as pending, waiting with origin/binding, done with binding, or failed. Project waiting origins as `waiting:pending` and `waiting:failed`; preserve the chosen binding and origin across interruption instead of inferring completion from an object.
|
|
26
27
|
- A zero-turn Claude session has no transcript, so it can never be resumed: every identity gate that compares session ids (restore, coordinator verify, daemon readiness) must tolerate a fresh session while the transcript is absent and stay strict the moment one exists. The commit snapshot records `sessionPersisted` for the restored daemon, and snapshots too old to have it are re-derived from disk. Never treat "session id changed" as proof of a lost conversation without checking the transcript first. A post-restore detachment repair (#463) is bound to its last accepted session even when its transcript is absent; never use the zero-turn exception to replace that recorded conversation.
|
|
@@ -33,9 +34,10 @@ Scope: the operator console, control WS and its protocol, state files, the dashb
|
|
|
33
34
|
- Four checks read another release's source text for an exact line: `coordinatorCurrent` (`export async function disposeRecovery` in `src/cli/upgrade.ts`), `targetReadsWaivers` (`export function readRecoveryWaivers` in `src/hub/restart.ts`), `coordinatorRestarts` (`export function recordRestart` in `src/cli/upgrade.ts`) and `supportsAction` (`export async function reattachRecovery` in `src/cli/upgrade.ts`). Renaming, moving or un-exporting one makes this release read as an older one: keep those lines until a capability list replaces the checks. The reattach CLI dispatch must use the same `supportsAction` check as its printed command; never redirect a fallback command back to a coordinator without that action.
|
|
34
35
|
- `reattachRecovery` (`src/cli/upgrade.ts`, #463) relaunches only the peer a blocked receipt's `blockedReattachment` names, once per its `reattachments` marker, after the lock, the runner claim and a re-read, and only against the same held target instance and version. The terminal proof is `inspectSavedTerminal` (`src/cli/terminal-recovery.ts`): only `orphaned` or `missing` (a typed stale handle and a complete, untruncated inventory) counts as gone; `unknown`, `changed` or a disconnected-only reading never authorizes a terminal. It counts as reattached only on a `done` restore receipt with the exact recorded session id, session file and project root, then leaves the operation blocked for `recovery resume`. It is person-only: `classifyPeerCommand` and `liveTestRefusal` refuse it from an agent shell.
|
|
35
36
|
- The interactive screens (`src/cli/upgrade-interactive.ts`, #272) change an operation only by running an existing command (`recovery resume|abort|dispose|reattach`, `reset`) through `io.run`, and build their menu from `nextChoices`, whose text is `nextActions`: never call `abortRecovery`, `disposeRecovery` or a driver from a screen, and never offer a choice `next` does not have. The one addition is `cancellableWait` (a runner still at `prepare` with no effect): read the receipt again at the key press, stop only a runner `signedRunner` names (a claim without a matching process signature is never signalled), then run `recovery abort`, which decides again. Agents are ended (`endPlannedPeer`) only before an operation holds the lock, and only after the lock, the hub instance, the attached session and the terminal's handle and incarnation were read again: never on the plan as it was drawn. Ctrl+C leaves the screens and the terminal is not read after it; a prompt answers an interrupt with null, never text. What a hub says it waits for (`recovery.waiting`, the daemon's `blockers`) is shown, never decided on.
|
|
36
|
-
- Recovery presentation (#466, `src/cli/upgrade-presentation*.ts`) only renders, and only when `humanRecovery` holds (a TTY and no `--json`); `--json` stays the raw document. Recognise `--json`, `--verbose` and `--color=` only through `recoveryPresentationFlag` (`src/cli/recovery-presentation-options.ts`), which `liveTestRefusal` and `classifyPeerCommand` share, so a presentation flag cannot hide reattach or a repaired dispose from authorization. A screen's Enter default (`preferred`) is read-only (refresh, follow the runner, read the receipt); apply, resume, reattach and destructive choices need their own key, and an item with a `disabledReason` never runs. The receipt's `waits` and `presentationCause` describe the current attempt only: written best-effort (a failed timing save warns, never fails the operation or replaces the wait's own error) and cleared at each resume. Without them show `timing unknown` or an unknown cause; never substitute `updatedAt` or guess a cause from error text. `endInspection` (the completion and stop-and-archive summary) only inspects, and a failed reading is `runtime unknown`.
|
|
37
|
+
- Recovery presentation (#466, `src/cli/upgrade-presentation*.ts`) only renders, and only when `humanRecovery` holds (a TTY and no `--json`); `--json` stays the raw document. Recognise `--json`, `--verbose` and `--color=` only through `recoveryPresentationFlag` (`src/cli/recovery-presentation-options.ts`), which `liveTestRefusal` and `classifyPeerCommand` share, so a presentation flag cannot hide reattach or a repaired dispose from authorization. A screen's Enter default (`preferred`) is read-only (refresh, follow the runner, read the receipt); apply, resume, reattach and destructive choices need their own key, and an item with a `disabledReason` never runs. The receipt's `waits` and `presentationCause` describe the current attempt only: written best-effort (a failed timing save warns, never fails the operation or replaces the wait's own error) and cleared at each resume. Without them show `timing unknown` or an unknown cause; never substitute `updatedAt` or guess a cause from error text. With no typed cause, `renderBlocked` may show only the first recorded error line through `terminalText`, controls stripped, capped at 512 code points and labelled `typed cause unknown`, with the uniform action; never classify it or choose an option from its text (#478). `endInspection` (the completion and stop-and-archive summary) only inspects, and a failed reading is `runtime unknown`.
|
|
37
38
|
- `upgrade()` (`src/cli/main.ts`, #272) never coordinates a target newer than its own `VERSION`: it hands over to that release's coordinator (`bun x --package @staix/agent-hub@<to> ahub upgrade --to <to>` with the same flags), and `AGENTHUB_UPGRADE_HANDOVER=1` keeps the process it started from handing over again. `--to` passes `exactVersion` before it reaches a package spec or the registry. `latestRelease` supplies it only for a terminal run or a dry run: `--yes`, and an apply through a pipe, still name the release.
|
|
38
39
|
- Recovery judges the registry claim as it is now (`inspectRecovery` re-reads `registeredProjects()`), never the plan's frozen copy; a registry that cannot be read is not inspected, never stopped (#226).
|
|
40
|
+
- `inspectRecovery` catches only `RecoveryDeferredHubError` from `readRestartSnapshot`: keep the stopped state and return the static `snapshotError` blocker without a `snapshot` binding. Never echo rejected content, repair the snapshot or offer resume from it (#478).
|
|
39
41
|
- The managed Claude hook (`src/cli/facts-hook.ts`) may only add `permissionDecision: "allow"` on `PreToolUse`, and for `ask-when-needed` only what `projectFileTool` accepts: a hub that is down, slow or refuses decides nothing. The daemon puts `permission` in a `facts` reply only for the hook of this instance's recorded `ahub claude` launch (the same `launchId`, not `--unattended`), and a permission-only hook (`hookPurpose`) never drives busy state (#242).
|
|
40
42
|
- `ahub claude` (`src/cli/launch.ts`, #270) publishes `claude-launch.json` for every launch against a running hub, atomically and without a lock: hooks, permission status, session observation and recovery read its `launchId`, so never make it wait for native-proof state. The native child's pid and signature go only into the private `claude-native-<binding hash>.json`, keyed by the exact launch record, written once under `claude-launch.lock` and removed at that child's own exit; never put them in `claude-launch.json`, where a late callback of an earlier launch could replace a newer record. Crash cleanup of a previous launch's settings file (`cleanupStaleClaudeSettings`) needs the launcher gone by `processLiveness` and that native proof, and for a `direct-child` proof the native pid absent from an available `processTable`: a direct child may exec in place, so its signature certifies nothing. A script or interpreter gets no proof and keeps its settings file.
|
|
41
43
|
|
|
@@ -5,7 +5,7 @@ Scope: the task flow, assignment, the state machine and the hub's MCP tools. Rea
|
|
|
5
5
|
- Tool callers are models: MCP `inputSchema` is not enforced on the way in. Normalize at the boundary (`cleanRefs`) before anything reaches the board, and never throw after a board write: a publish or notify that follows a `review()` or `done()` board write goes through the one `releaseNotice` guard, which turns a failed publish into a console notice, drops a failed notify and never retries (#231, #243). An assignment's delivery once its board write is saved (#276, #297): the move is fact, the failed publish is not retried in place (the journal's own recovery covers a storage error, and a retry may duplicate a publish that actually left), and a person looks for a queued or needs_review delivery with `ahub queue list --peer <owner>` (then `ahub queue show <id>`, or the accepted entry in `ahub task show <id>` if the journal is down) before re-sending with `ahub task assign <id> <owner>`. The distinction is made once, in `assignOwner`: a failure after the write prints the one console notice from `undeliveredNotice` right there, once, and throws `AssignmentUndeliveredError` carrying the saved task; callers (`review()`'s repeated rejection, `escalate` and its callers, the ready offer, both relays, the idle sweep, rescreen) decide control flow by the type and print nothing more, and `propose`, `assignTo` and `decline` additionally let the error reach the caller in its turn, which is a person or peer that can act. Two exceptions get no notice, by construction: a row superseded meanwhile, meaning a later history entry that moves or settles the task (another assignment, an unassign, the owner's accept, done, approval, close), while entries that do neither (the hub's quiet bookkeeping: answers, reviewer changes, idle sweeps, check results) leave the assignment undelivered and the notice stands, so the original error goes to the caller as before only on supersession; and a claim's own accepted write and "claimed by" line stay outside the delivery rule (a claim is its own hand-over; its overlap and routing announcements to earlier owners are deliveries and share the wrapper, so a failure there does read as one). A failure before the write stays a plain `Error` (#254), a failed write leaves the ready offer unconsumed (#231), and the notices before the write (waits, no-owner, the reserved-owner line) go through `releaseNotice` too, so those pre-write notices cannot fail their caller.
|
|
6
6
|
- Post-await assignment publication is fenced by current role authority: supersession returns the current task without a saved-delivery notice; reviewer-only changes refresh current metadata and do not suppress a valid owner delivery. A replacement notice goes only to a role that actually received the request, not to an unread-only queue.
|
|
7
7
|
- Console task row assignment/unassignment carries the exact role, task id and `flowRevision` (history length), checks it before effects and passes it to `Board.update`; a changed revision requires reload. Person owner removal clears owner/reservation and suppresses automatic offers until a later explicit person assignment/reservation; refuse owner removal under review or after approval, and preserve the required PII person reviewer.
|
|
8
|
-
- Role withdrawal matches the task id, role kind and owner-request header. Owner acceptance evidence must be that peer's accepted entry after the latest ownership event, never another owner's state or an earlier assignment span. Withdraw unread requests without inventing delivery evidence. Preserve the original former-owner stop notice after a superseding escalation, while suppressing stale success/decision tails.
|
|
8
|
+
- Role withdrawal matches the task id, role kind and owner-request header. Owner acceptance evidence must be that peer's accepted entry after the latest ownership event, never another owner's state or an earlier assignment span. Withdraw unread requests without inventing delivery evidence. Find them with `bus.pendingEnvelopes(peer)`, which includes recovery-deferred notices, never the snapshot's queues (#478). Preserve the original former-owner stop notice after a superseding escalation, while suppressing stale success/decision tails.
|
|
9
9
|
- `assign()` never defaults to the task's current owner, or a decline can only come back to the decliner. A reservation (#207) is the same risk: the reserved owner goes first only through `blocked()`, and a decline or an escalation for reason `rejections` or `manual` records the owner it left in the history entry's `from`, which `excluded()` reads, so no later reroute hands the task back through its reservation.
|
|
10
10
|
- The state machine allows `in_progress -> approved` only for classes without a reviewer; `review()` checks `in_review` itself.
|
|
11
11
|
- Treat `cancelled` as closed through `isClosedTaskState`, but keep dependency release restricted to approval. Cancellation goes through `Tasks.cancel`, preserves the row and roles, clears its reservation/cohort, sends revision-bound stop notices and writes no outcome memory; never map it to approval or cancel a native turn or worktree. Planner cancellation stays limited to its own historically unaccepted proposed/backlog work; accepted work remains person-only.
|
package/docs/operations.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Operations guide
|
|
2
2
|
|
|
3
|
-
This guide describes ahub 0.12.
|
|
3
|
+
This guide describes ahub 0.12.33 and control protocol 16. Live verification
|
|
4
4
|
results and remaining prerequisites are recorded separately in [the smoke ledger](smoke.md).
|
|
5
5
|
|
|
6
6
|
## Agent-operated live verification
|
|
@@ -1837,6 +1837,9 @@ The runtime reset (the default):
|
|
|
1837
1837
|
through the delivery journal, one `resolution_history` entry each. Work
|
|
1838
1838
|
that was dispatching or accepted when the hub stopped is held as
|
|
1839
1839
|
`needs_review` first, so it is discarded too;
|
|
1840
|
+
- discards hub notices a recovery hold kept back the same way, one entry per
|
|
1841
|
+
recipient, without delivering them; the dry run lists them as
|
|
1842
|
+
`q:<peer>:<id>`;
|
|
1840
1843
|
- clears manual peer holds (`ahub pause`), budget pauses and conductor holds;
|
|
1841
1844
|
- drops the agent session resume pointers `sessions.json`,
|
|
1842
1845
|
`claude-session.json` and `claude-context.json`, so the next launch starts
|
|
@@ -1949,21 +1952,21 @@ section is what they run.
|
|
|
1949
1952
|
|
|
1950
1953
|
Upgrade running projects with the target release's own coordinator. It accepts
|
|
1951
1954
|
a running source on control protocol 9 (0.6.x), 10 (0.7.0 through 0.12.0),
|
|
1952
|
-
11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 through 0.12.
|
|
1955
|
+
11 (0.12.1 and 0.12.2), 12 (0.12.3), 13 (0.12.4 through 0.12.15), 14 (0.12.16), 15 (0.12.17 through 0.12.19) or 16 (0.12.20 through 0.12.33), and only
|
|
1953
1956
|
a target on its own protocol, so the target's coordinator fits every supported
|
|
1954
1957
|
source and carries every recovery fix released up to it. Protocol 8 and older
|
|
1955
1958
|
(0.5.x and earlier) are refused as `manual-bootstrap-required`. Run from the
|
|
1956
1959
|
project directory, without replacing the global CLI first:
|
|
1957
1960
|
|
|
1958
1961
|
```bash
|
|
1959
|
-
bunx --package @staix/agent-hub@0.12.
|
|
1960
|
-
bunx --package @staix/agent-hub@0.12.
|
|
1962
|
+
bunx --package @staix/agent-hub@0.12.33 ahub upgrade --to 0.12.33 --dry-run
|
|
1963
|
+
bunx --package @staix/agent-hub@0.12.33 ahub upgrade --to 0.12.33 --yes
|
|
1961
1964
|
```
|
|
1962
1965
|
|
|
1963
1966
|
| Running now | Coordinator to use |
|
|
1964
1967
|
| --- | --- |
|
|
1965
1968
|
| 0.6.x (protocol 9) | the target's, through `bunx` as above |
|
|
1966
|
-
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 through 0.12.
|
|
1969
|
+
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 through 0.12.15 (protocol 13), 0.12.16 (protocol 14), 0.12.17 through 0.12.19 (protocol 15), 0.12.20 through 0.12.33 (protocol 16) | the target's, through `bunx` as above |
|
|
1967
1970
|
| any supported source, with the installed CLI already at the target | `ahub upgrade` below, which is the same coordinator |
|
|
1968
1971
|
| 0.5.x or earlier (protocol 8 and older) | not supported: bootstrap by hand with the matching CLI |
|
|
1969
1972
|
|
|
@@ -1995,14 +1998,14 @@ projects first:
|
|
|
1995
1998
|
|
|
1996
1999
|
```bash
|
|
1997
2000
|
ahub restart --dry-run
|
|
1998
|
-
ahub upgrade --to 0.12.
|
|
2001
|
+
ahub upgrade --to 0.12.33 --dry-run
|
|
1999
2002
|
```
|
|
2000
2003
|
|
|
2001
2004
|
Apply only after reviewing the plan:
|
|
2002
2005
|
|
|
2003
2006
|
```bash
|
|
2004
2007
|
ahub restart --yes
|
|
2005
|
-
ahub upgrade --to 0.12.
|
|
2008
|
+
ahub upgrade --to 0.12.33 --yes
|
|
2006
2009
|
ahub recovery status [<operation-id>]
|
|
2007
2010
|
ahub recovery resume [<operation-id>]
|
|
2008
2011
|
ahub recovery abort [<operation-id>]
|
|
@@ -2029,6 +2032,15 @@ line of its own to `hub.log`), and until no task command or dashboard action is
|
|
|
2029
2032
|
still running. A check the commit's stop kills writes to the board after the
|
|
2030
2033
|
commit has recorded it, and the operation stays blocked.
|
|
2031
2034
|
|
|
2035
|
+
From 0.12.33, while an operation holds a source of that release or later, and
|
|
2036
|
+
while a restored target waits for release, the notices the hub writes itself
|
|
2037
|
+
(other than `[FYI]` ones) are kept apart from the carried queues, durably in
|
|
2038
|
+
the delivery journal and the restart snapshot, so the queue comparison still
|
|
2039
|
+
matches exactly. Release queues them in order and saves each move before
|
|
2040
|
+
anything is delivered. A target that does not advertise `recovery_deferred_hub`
|
|
2041
|
+
gets them in its ordinary queues before commit instead. At most 2048 such
|
|
2042
|
+
notices wait; a further one is refused, and none already kept is dropped.
|
|
2043
|
+
|
|
2032
2044
|
0.9.0 turns on for every project with a config file, whether or not it has the
|
|
2033
2045
|
block: `limits` (12 messages a minute per sender, 6 per recipient, 6
|
|
2034
2046
|
`[IMPORTANT]` an hour, 120 s repeats) and `budget.wait_max_min` (30). Set them
|
|
@@ -2196,7 +2208,12 @@ and fresh-session choices carry a warning and name the consequence before the
|
|
|
2196
2208
|
person confirms. Raw receipt keys and the original recorded error are available
|
|
2197
2209
|
with `j` or `--verbose`. The four preservation failures name the actual failed
|
|
2198
2210
|
queue, manual-pause, board or budget comparison; an older untyped cause remains
|
|
2199
|
-
unknown rather than being guessed from its text.
|
|
2211
|
+
unknown rather than being guessed from its text. When nothing more specific
|
|
2212
|
+
applies (no typed cause, reattachment or `database_busy`), Why shows the first
|
|
2213
|
+
line of the recorded error, with control characters removed and cut at 512
|
|
2214
|
+
characters, marked `typed cause unknown`. What to do then says to resolve that
|
|
2215
|
+
error, then resume or choose an offered action. Nothing is classified from that
|
|
2216
|
+
text, and JSON and piped output are unchanged.
|
|
2200
2217
|
|
|
2201
2218
|
Completion and stop-and-archive use a human summary on a TTY. It separates the
|
|
2202
2219
|
outcome, recorded plugin/global installation effects, remaining work and the
|
|
@@ -2256,7 +2273,10 @@ each receipt allows, given what is live, is tabulated in the recovery spec
|
|
|
2256
2273
|
(`docs/specs/2026-09-20-upgrade-recovery-design.md`, "Receipts, evidence and
|
|
2257
2274
|
next actions").
|
|
2258
2275
|
|
|
2259
|
-
- The source hold lapses after 10 minutes.
|
|
2276
|
+
- The source hold lapses after 10 minutes. If releasing it fails because the
|
|
2277
|
+
source's delivery storage fails, the hold stays and the expiry is retried
|
|
2278
|
+
every second until storage recovers; restore the storage before `abort`.
|
|
2279
|
+
A later `resume` prepares the same
|
|
2260
2280
|
source again and checks its peers again, keeping the receipts, so no terminal
|
|
2261
2281
|
is closed twice. It refuses, naming the next step, when the source daemon was
|
|
2262
2282
|
replaced, when another operation holds it, or when a peer changed: a peer whose
|
|
@@ -2318,7 +2338,11 @@ next actions").
|
|
|
2318
2338
|
that session. A target that dies again in the same `resume` stops it with the
|
|
2319
2339
|
restart count; look at its `hub.log` before resuming again. Without the
|
|
2320
2340
|
snapshot, or for an operation started by a coordinator older than 0.12.21,
|
|
2321
|
-
the way out is stop-and-archive.
|
|
2341
|
+
the way out is stop-and-archive. A snapshot whose kept hub notices fail
|
|
2342
|
+
validation blocks that restart with `restart snapshot deferred hub notices
|
|
2343
|
+
are invalid; restore is blocked`. The hub still reads as stopped, so
|
|
2344
|
+
stop-and-archive (and abort, where it applies) stays offered; nothing
|
|
2345
|
+
repairs the snapshot or offers to resume from it.
|
|
2322
2346
|
- A Codex conversation comes back only when a rollout file naming its thread
|
|
2323
2347
|
exists under `sessions/` of the store the restored terminal uses (the
|
|
2324
2348
|
recorded `CODEX_HOME`, else `~/.codex`). Codex writes that file with the
|
|
@@ -2648,7 +2672,7 @@ ahub task cancel <id> --reason <text> [--superseded-by <ref>] [--revision N]
|
|
|
2648
2672
|
|
|
2649
2673
|
The console also offers cancellation with the selected task's flow revision. Approved tasks cannot be cancelled. The planner's `hub_task_cancel` tool is limited to its own unaccepted backlog/proposed work; previously accepted work requires the person. Cancellation sends the former owner/reviewer a stop notice, releases overlap/reservation participation and keeps dependents blocked with a cancelled-dependency notice. It does not stop a native turn, delete a worktree or write an outcome memory note. Cancelled work is closed history, counted separately from approved work; it does not satisfy an approval dependency. A still-running private turn keeps its privacy fence after cancellation. Use the displayed revision to avoid acting on a task that changed while it was selected.
|
|
2650
2674
|
|
|
2651
|
-
The daemon advertises `task_cancel`; a current CLI refuses cancellation against older daemons. Recovery retains verified target features beside its immutable root/digest, reuses them on resume and checks only sources not yet past commit; a legacy receipt without that evidence needs a target probe only before commit. Upgrade/restart refuses a target without this feature while cancelled rows exist and names their ids. No cancelled row is rewritten as approved. An old binary launched by hand remains unsupported: it can count cancelled work as open or fail to move it. Use a compatible coordinator instead. Live task cancellation and sweep enablement are separate post-release operations.
|
|
2675
|
+
The daemon advertises `task_cancel`; a current CLI refuses cancellation against older daemons. Recovery retains verified target features beside its immutable root/digest, reuses them on resume and on each lease renewal before a terminal is closed, and checks only sources not yet past commit; a legacy receipt without that evidence needs a target probe only before commit. Upgrade/restart refuses a target without this feature while cancelled rows exist and names their ids. No cancelled row is rewritten as approved. An old binary launched by hand remains unsupported: it can count cancelled work as open or fail to move it. Use a compatible coordinator instead. Live task cancellation and sweep enablement are separate post-release operations.
|
|
2652
2676
|
|
|
2653
2677
|
## Task idle sweep
|
|
2654
2678
|
|
package/docs/quickstart.md
CHANGED
|
@@ -17,7 +17,7 @@ ahub setup # installs the Claude Code channel plu
|
|
|
17
17
|
Or use the matching GitHub release:
|
|
18
18
|
|
|
19
19
|
```bash
|
|
20
|
-
bun add -g github:STAIxBWLB/agent-hub#v0.12.
|
|
20
|
+
bun add -g github:STAIxBWLB/agent-hub#v0.12.33
|
|
21
21
|
ahub setup
|
|
22
22
|
```
|
|
23
23
|
|
|
@@ -30,9 +30,9 @@ and upgrade or crash recovery, see the [operations guide](operations.md). To
|
|
|
30
30
|
move running projects to a newer release later, run `ahub upgrade` in a
|
|
31
31
|
terminal: it reviews the plan with you, applies it and follows it.
|
|
32
32
|
|
|
33
|
-
## Upgrading to 0.12.
|
|
33
|
+
## Upgrading to 0.12.33
|
|
34
34
|
|
|
35
|
-
The upgrade runs with the target release's own coordinator. If the operation blocks because a restored Codex, Claude or Pi terminal disappeared, the person runs `ahub recovery reattach <operation-id> codex|claude|pi` to relaunch that peer with its recorded session (one relaunch per receipt), then `ahub recovery resume` to finish.
|
|
35
|
+
The upgrade runs with the target release's own coordinator, so an upgrade from 0.12.32 with cancelled tasks and a managed terminal, or with a notice the target sends right after it starts, no longer blocks (#478). If the operation blocks because a restored Codex, Claude or Pi terminal disappeared, the person runs `ahub recovery reattach <operation-id> codex|claude|pi` to relaunch that peer with its recorded session (one relaunch per receipt), then `ahub recovery resume` to finish.
|
|
36
36
|
|
|
37
37
|
From 0.12.30 or earlier, existing Jcode project MCP entries need one refresh before conductor or dashboard starts. Once per project, the person runs:
|
|
38
38
|
|
package/docs/security.md
CHANGED
|
@@ -119,7 +119,12 @@ The operator uses a plain terminal or `ahub console`. A Claude `!` command that
|
|
|
119
119
|
inherits the agent markers follows the same rule. The console strips control
|
|
120
120
|
sequences from agent and daemon text before it quotes forged headers, and its
|
|
121
121
|
colors come only from a fixed palette applied afterwards, so message text cannot
|
|
122
|
-
set styles or operate the terminal.
|
|
122
|
+
set styles or operate the terminal. The recovery blocked screen shows a recorded
|
|
123
|
+
error only when no typed cause exists: its first line, passed through the
|
|
124
|
+
console's terminal sanitizer with control characters removed, capped at 512
|
|
125
|
+
code points and marked as an untyped recorded error. A restart snapshot whose
|
|
126
|
+
kept hub notices fail validation is reported with a fixed message, never with
|
|
127
|
+
its content.
|
|
123
128
|
|
|
124
129
|
This is an honest default against accidental impersonation and injected commands.
|
|
125
130
|
It does not make the token inaccessible to an agent with unrestricted project
|
|
@@ -849,3 +849,23 @@ Recovery authorization and the presentation parser share the pure
|
|
|
849
849
|
skipping the accepted one-shot presentation flags (`--json`, `--verbose`, `--color=<mode>`). Such flags never hide
|
|
850
850
|
person-only reattachment from agent or live-test refusal checks; normalization
|
|
851
851
|
confers no authority and unknown flags still fail the command parser.
|
|
852
|
+
|
|
853
|
+
|
|
854
|
+
## Recovery-held hub publication amendment (#478)
|
|
855
|
+
|
|
856
|
+
2026-10-18: Keep exact carried-queue integrity equality. Every checked prepare renewal carries the verified target features stored on the operation. Ordinary post-commit observational inspection stays free of pre-commit compatibility checks.
|
|
857
|
+
|
|
858
|
+
Only an authenticated recovery operation with verified target support opts into hub deferral; a bare bus delivery fence retains its existing queue admission and does not confer daemon operation authority. The bus accepts hub-originated, deliverable notices during recovery into optional `BusSnapshot.deferredHub`, persisted by the existing private delivery journal and restart snapshot. These originals are outside delivery queues and their integrity projection. The accepted audience and envelope metadata, including private marking, are captured without new text-writing paths. FYI and hop-dropped publications keep their existing record-only behavior. Withdrawal and role/feed revocation also cover deferred originals. Backlog retention counts them as accepted work.
|
|
859
|
+
|
|
860
|
+
A target advertises `recovery_deferred_hub` beside `task_cancel`. Before a commit to a target without that verified feature, the source materializes deferred originals into normal queues; expected integrity therefore verifies those rows normally. No old target is asked to preserve a field it cannot read.
|
|
861
|
+
|
|
862
|
+
Deferred restore validation refuses malformed, non-hub, duplicate or already-accepted ids with a typed recovery error. The FIFO is capped at 2048 originals; overflow refuses admission without dropping accepted work or releasing the hold. Each release transfers an original through normal publication and checkpoints its removal from the deferred FIFO together with its queued/seen state before delivery is allowed. A crash resumes from the journal, preserving id dedupe. Failed checkpoints leave the original pending and recovery held. A retry of an already archived release still completes the bus release; a later ordinary startup also completes durable pending publication. Abort and lease expiry retain their recovery ownership until pending publication can be checkpointed.
|
|
863
|
+
|
|
864
|
+
When no typed presentation cause exists, the human blocked screen shows only the existing sanitized first recorded error line, with controls removed and a 512-code-point cap. The action is uniform: resolve the recorded error, then resume or choose an offered action. Feature compatibility is a conditional option only, never classified from an error substring. JSON and piped rendering are unchanged.
|
|
865
|
+
|
|
866
|
+
|
|
867
|
+
### Deferred-notice edge paths (#478 review amendment)
|
|
868
|
+
|
|
869
|
+
An expired source lease retries a failed release checkpoint on a short timer; a vanished coordinator is not required to unstick the hold once storage recovers. Reset previews count deferred recipients as pending delivery ids, and stopped-runtime reset atomically records their discard audit and removes the originals without publication. Deferred records require their captured recipient array. Flush rollback restores only publication-owned state (affected queue arrays in place, deferred and seen ids, withdrawal bookkeeping and pending outcomes); it never reloads a live bus snapshot or changes its recovery mode, prefaces or independent pauses. Advisory nudges, like publication events, run after the checkpoint.
|
|
870
|
+
|
|
871
|
+
A corrupt deferred field still refuses daemon restore. Coordinator inspection catches only the typed deferred-corruption error, keeps independently verified stopped evidence, reports an unknown snapshot binding with a static blocker, and leaves person abort/stop-and-archive exits available. It never claims that the corrupt snapshot can be resumed, repairs it or exposes rejected content.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# 0.12.33 release verification
|
|
2
|
+
|
|
3
|
+
This record covers release preparation and source review. It does not claim publication, installation, a running-hub restart or native/live acceptance.
|
|
4
|
+
|
|
5
|
+
## Candidate
|
|
6
|
+
|
|
7
|
+
Version 0.12.33, control protocol 16 (`src/hub/control-client.ts`), from merged main `fe53b272db455fd5b95b4a504e806a0a0625a26c`. The candidate includes the one PR merged since `v0.12.32`: #479 (issue #478). The exact first-party memory pin remains `@staix/agent-mem` 0.3.1.
|
|
8
|
+
|
|
9
|
+
- The upgrade runs with the target release's own coordinator (`bunx --package @staix/agent-hub@0.12.33 ahub upgrade`). An upgrade from 0.12.32 into 0.12.33 therefore already uses the fixed lease renewal and the deferred hub notices.
|
|
10
|
+
- #479 adds the target feature `recovery_deferred_hub` beside `task_cancel`. A target without it gets deferred notices in its ordinary queues before commit, so mixed-version upgrades keep exact queue verification.
|
|
11
|
+
- Still open: a real upgrade with a managed TUI and cancelled rows (#478), a real Orca upgrade exercising `ahub recovery reattach` (#463), the native real-terminal console settings leg (#439 AC5), installed managed-Pi #418 AC8 (#142) and #438 AC7.
|
|
12
|
+
|
|
13
|
+
## Document review
|
|
14
|
+
|
|
15
|
+
`node scripts/check-docs.mjs --require-fresh` reported 10 stale documents, all made stale by #479; editing `daemon.md` made an 11th, `docs/agent-notes/README.md`. Each was compared against the #479 source diff (`cf3026fe..fe53b272`). The limits and messages cited were checked in the source: `MAX_DEFERRED_HUB` is 2048 (`src/hub/deferred-hub.ts`), the blocked-screen cap is 512 code points and the label is `typed cause unknown` (`src/cli/upgrade-presentation.ts`), and the snapshot blocker message comes from `src/cli/upgrade-runtime.ts`. Version references were moved to 0.12.33 in the same pass. No document review is deferred.
|
|
16
|
+
|
|
17
|
+
| Document | Prior source stamp | Review result | Checked claims and correction |
|
|
18
|
+
| --- | --- | --- | --- |
|
|
19
|
+
| `README.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | Status 0.12.33; Jcode entry unchanged; the verification link and the open #478 live leg. `task_cancel` text still holds. |
|
|
20
|
+
| `docs/quickstart.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | `v0.12.33` install example and the upgrade section for the #478 fixes; reattach and resume text unchanged. |
|
|
21
|
+
| `docs/security.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | Blocked-screen recorded error passes the terminal sanitizer, controls stripped, capped; a corrupt snapshot is reported with a fixed message only. |
|
|
22
|
+
| `docs/operations.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | 0.12.33 header and upgrade examples; deferred hub notices during recovery and the downgrade path; the lease-expiry retry; the corrupt-snapshot blocker; the blocked-screen fallback; runtime reset discards deferred notices; features sent on lease renewal. |
|
|
23
|
+
| `docs/agent-notes/README.md` | `cf3026feab218207f0c21b300552eada0d5be227` | unchanged | Stale only because this pass edited `daemon.md`; the console exact-row navigation rule it explains is unchanged. |
|
|
24
|
+
| `docs/agent-notes/adapters.md` | `cf3026feab218207f0c21b300552eada0d5be227` | unchanged | #479 touched only recovery and bus wiring in `daemon.ts`; no adapter, spawn or approval rule changed. |
|
|
25
|
+
| `docs/agent-notes/budget.md` | `cf3026feab218207f0c21b300552eada0d5be227` | unchanged | Pause persistence and resume-notice order unchanged; "publish, then withdraw" holds, and withdraw now also covers deferred notices. |
|
|
26
|
+
| `docs/agent-notes/bus.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | Deferred notices stay outside queues and the integrity view, every pending-work read includes them, the cap refuses rather than drops; release checkpoints each move and a failed checkpoint restores only publication state. |
|
|
27
|
+
| `docs/agent-notes/daemon.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | Every prepare sends stored `targetFeatures`; when hub notices are deferred and the lease-expiry retry; `inspectRecovery` catches only `RecoveryDeferredHubError`; the `renderBlocked` fallback rule. |
|
|
28
|
+
| `docs/agent-notes/tasks.md` | `cf3026feab218207f0c21b300552eada0d5be227` | corrected | Role withdrawal finds unread requests through `bus.pendingEnvelopes`, which includes deferred notices. |
|
|
29
|
+
| `docs/agent-notes/tests.md` | `cf3026feab218207f0c21b300552eada0d5be227` | unchanged | The new tests follow the existing rules; `scripts/check.sh`, seeds and CI are unchanged. |
|
|
30
|
+
|
|
31
|
+
`AGENTS.md` area notes and `docs/verified.json` paths now include `src/hub/deferred-hub.ts` (bus and daemon) and `src/cli/reset.ts` (bus), the modules the new bus and daemon rules describe. A separate manifest commit restamps the reviewed documents against the reviewed source; it does not stamp itself. The release PR must use a merge commit, and committed freshness is rerun on main before tagging.
|
|
32
|
+
|
|
33
|
+
## Recorded checks
|
|
34
|
+
|
|
35
|
+
| PR | Final head | CI run |
|
|
36
|
+
| --- | --- | --- |
|
|
37
|
+
| [#479](https://github.com/STAIxBWLB/agent-hub/pull/479) | `e832901d10ab263e3887426534a6aebd3e8612ef` | [39085109057](https://github.com/STAIxBWLB/agent-hub/actions/runs/39085109057) |
|
|
38
|
+
|
|
39
|
+
Both OS check jobs, all four seeded shards and the `seeded guards` aggregate succeeded. The PR passed two OCR delegation review rounds by a Claude subagent: 21 of 21 files were accounted for, and the round-1 findings were fixed and verified at the final head. These prior PR gates do not replace the release candidate's own exact-head CI.
|
|
40
|
+
|
|
41
|
+
- Generated assets: `bun install --frozen-lockfile` and `bun run build` ran locally at `894350fa`. The only change to `plugins/agent-hub/server.js` and the plugin manifest is the version string 0.12.32 -> 0.12.33; the release PR's CI checks bundle freshness.
|
|
42
|
+
- The release PR's own exact-head CI and the release workflow's tag checks are recorded on the PR and the tag, not here.
|
package/docs/verified.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"schemaVersion": 1,
|
|
3
3
|
"documents": {
|
|
4
4
|
"README.md": {
|
|
5
|
-
"verifiedAgainst": "
|
|
5
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
6
6
|
"paths": [
|
|
7
7
|
"package.json",
|
|
8
8
|
"src/",
|
|
@@ -11,14 +11,14 @@
|
|
|
11
11
|
]
|
|
12
12
|
},
|
|
13
13
|
"docs/security.md": {
|
|
14
|
-
"verifiedAgainst": "
|
|
14
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
15
15
|
"paths": [
|
|
16
16
|
"src/",
|
|
17
17
|
"templates/"
|
|
18
18
|
]
|
|
19
19
|
},
|
|
20
20
|
"docs/operations.md": {
|
|
21
|
-
"verifiedAgainst": "
|
|
21
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
22
22
|
"paths": [
|
|
23
23
|
"package.json",
|
|
24
24
|
"src/",
|
|
@@ -35,7 +35,7 @@
|
|
|
35
35
|
]
|
|
36
36
|
},
|
|
37
37
|
"docs/quickstart.md": {
|
|
38
|
-
"verifiedAgainst": "
|
|
38
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
39
39
|
"paths": [
|
|
40
40
|
"package.json",
|
|
41
41
|
"src/",
|
|
@@ -43,7 +43,7 @@
|
|
|
43
43
|
]
|
|
44
44
|
},
|
|
45
45
|
"docs/agent-notes/adapters.md": {
|
|
46
|
-
"verifiedAgainst": "
|
|
46
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
47
47
|
"paths": [
|
|
48
48
|
"src/adapters/",
|
|
49
49
|
"src/pi/",
|
|
@@ -74,7 +74,7 @@
|
|
|
74
74
|
]
|
|
75
75
|
},
|
|
76
76
|
"docs/agent-notes/budget.md": {
|
|
77
|
-
"verifiedAgainst": "
|
|
77
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
78
78
|
"paths": [
|
|
79
79
|
"src/hub/budget.ts",
|
|
80
80
|
"src/cli/statusline-tee.ts",
|
|
@@ -86,24 +86,27 @@
|
|
|
86
86
|
]
|
|
87
87
|
},
|
|
88
88
|
"docs/agent-notes/bus.md": {
|
|
89
|
-
"verifiedAgainst": "
|
|
89
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
90
90
|
"paths": [
|
|
91
91
|
"src/hub/bus.ts",
|
|
92
92
|
"src/hub/envelope.ts",
|
|
93
93
|
"src/hub/limits.ts",
|
|
94
94
|
"src/hub/delivery-journal.ts",
|
|
95
|
+
"src/hub/deferred-hub.ts",
|
|
95
96
|
"src/hub/inference.ts",
|
|
96
97
|
"src/hub/daemon.ts",
|
|
97
|
-
"src/hub/supervision.ts"
|
|
98
|
+
"src/hub/supervision.ts",
|
|
99
|
+
"src/cli/reset.ts"
|
|
98
100
|
]
|
|
99
101
|
},
|
|
100
102
|
"docs/agent-notes/daemon.md": {
|
|
101
|
-
"verifiedAgainst": "
|
|
103
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
102
104
|
"paths": [
|
|
103
105
|
"src/hub/daemon.ts",
|
|
104
106
|
"src/hub/control-client.ts",
|
|
105
107
|
"src/adapters/claude-channel.ts",
|
|
106
108
|
"src/hub/restart.ts",
|
|
109
|
+
"src/hub/deferred-hub.ts",
|
|
107
110
|
"src/hub/recovery-store.ts",
|
|
108
111
|
"src/hub/snapshots.ts",
|
|
109
112
|
"src/hub/manager.ts",
|
|
@@ -159,7 +162,7 @@
|
|
|
159
162
|
]
|
|
160
163
|
},
|
|
161
164
|
"docs/agent-notes/tasks.md": {
|
|
162
|
-
"verifiedAgainst": "
|
|
165
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
163
166
|
"paths": [
|
|
164
167
|
"src/hub/tasks.ts",
|
|
165
168
|
"src/hub/board.ts",
|
|
@@ -177,7 +180,7 @@
|
|
|
177
180
|
]
|
|
178
181
|
},
|
|
179
182
|
"docs/agent-notes/tests.md": {
|
|
180
|
-
"verifiedAgainst": "
|
|
183
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
181
184
|
"paths": [
|
|
182
185
|
"test/",
|
|
183
186
|
"scripts/check.sh",
|
|
@@ -196,7 +199,7 @@
|
|
|
196
199
|
]
|
|
197
200
|
},
|
|
198
201
|
"docs/agent-notes/README.md": {
|
|
199
|
-
"verifiedAgainst": "
|
|
202
|
+
"verifiedAgainst": "38dfe1876c3b20d061b6232919e28965e1bb486e",
|
|
200
203
|
"paths": [
|
|
201
204
|
"src/cli/console-state.ts",
|
|
202
205
|
"src/cli/attention.ts",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-hub",
|
|
3
|
-
"version": "0.12.
|
|
3
|
+
"version": "0.12.33",
|
|
4
4
|
"description": "Channel between Claude Code and the agent-hub daemon: peer messages from Codex, Kimi and the local worker arrive as channel events; hub_send replies.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Young Joon Lee",
|
|
@@ -16431,7 +16431,7 @@ import { join as join6 } from "path";
|
|
|
16431
16431
|
// package.json
|
|
16432
16432
|
var package_default = {
|
|
16433
16433
|
name: "@staix/agent-hub",
|
|
16434
|
-
version: "0.12.
|
|
16434
|
+
version: "0.12.33",
|
|
16435
16435
|
description: "Native multi-agent hub: Claude Code, Codex, Kimi Code, Pi and local inference as peers in one project",
|
|
16436
16436
|
license: "MIT",
|
|
16437
16437
|
type: "module",
|
package/src/cli/reset.ts
CHANGED
|
@@ -4,6 +4,7 @@ import { chmodSync, copyFileSync, existsSync, lstatSync, mkdirSync, readdirSync,
|
|
|
4
4
|
import { join } from "node:path";
|
|
5
5
|
import { Bus } from "../hub/bus.ts";
|
|
6
6
|
import { DeliveryJournal } from "../hub/delivery-journal.ts";
|
|
7
|
+
import { validateDeferredHub } from "../hub/deferred-hub.ts";
|
|
7
8
|
import type { Project } from "../hub/registry.ts";
|
|
8
9
|
import { HUB_DB_BUSY_TIMEOUT_MS } from "../hub/db-timeout.ts";
|
|
9
10
|
|
|
@@ -73,7 +74,14 @@ export function planReset(stateDir: string, projectId: string): ResetPlan {
|
|
|
73
74
|
const bus: unknown = JSON.parse(meta?.bus_snapshot || "{}"), manual: unknown = JSON.parse(meta?.manual_paused || "[]");
|
|
74
75
|
shaped(isList(manual, isText), "manual_paused");
|
|
75
76
|
shaped(isObject(bus) && (bus.manualPaused === undefined || isList(bus.manualPaused, isText)) && (bus.queues === undefined || (isObject(bus.queues) && Object.values(bus.queues).every((queue) => isList(queue, isEnvelope)))), "bus_snapshot");
|
|
76
|
-
const { queues = {}, manualPaused = []
|
|
77
|
+
const { queues = {}, manualPaused = [], seen = [], deferredHub } = bus as {
|
|
78
|
+
queues?: Record<string, { id: string }[]>; manualPaused?: string[]; seen?: { id: string }[]; deferredHub?: unknown;
|
|
79
|
+
};
|
|
80
|
+
shaped(isList(seen, isEnvelope), "bus_snapshot.seen");
|
|
81
|
+
// Deferred notices are pending delivery too, but must not be published just to preview or reset them.
|
|
82
|
+
const deferred = validateDeferredHub(deferredHub, undefined, new Set([
|
|
83
|
+
...seen.map(env => env.id), ...Object.values(queues).flat().map(env => env.id),
|
|
84
|
+
]));
|
|
77
85
|
plan.manualHolds = [...new Set([...manual as string[], ...manualPaused])].sort();
|
|
78
86
|
// As Bus.queueList: a queued envelope that no open row stands for is listed as q:<peer>:<envelope id>.
|
|
79
87
|
// ponytail: the rule is copied so the dry run never opens the journal for writing; the reset tests compare the
|
|
@@ -84,6 +92,9 @@ export function planReset(stateDir: string, projectId: string): ResetPlan {
|
|
|
84
92
|
return (originals as { id: string }[]).map((env) => `${row.peer}:${env.id}`);
|
|
85
93
|
}));
|
|
86
94
|
for (const [peer, queue] of Object.entries(queues)) for (const env of queue) if (!recorded.has(`${peer}:${env.id}`)) plan.queued.push(`q:${peer}:${env.id}`);
|
|
95
|
+
for (const env of deferred) for (const peer of new Set(env.to)) {
|
|
96
|
+
if (!recorded.has(`${peer}:${env.id}`)) plan.queued.push(`q:${peer}:${env.id}`);
|
|
97
|
+
}
|
|
87
98
|
} finally { db.close(); }
|
|
88
99
|
return plan;
|
|
89
100
|
}
|
|
@@ -114,6 +125,7 @@ export function resetRuntime(stateDir: string, project: Project): { settled: str
|
|
|
114
125
|
const journal = new DeliveryJournal({ file, projectRoot: project.root, projectId: project.id, instanceId: randomUUID() });
|
|
115
126
|
const bus = new Bus({ journal });
|
|
116
127
|
try {
|
|
128
|
+
settled.push(...bus.discardDeferredHub("reset"));
|
|
117
129
|
for (const { id } of bus.queueList().filter((row) => row.state === "queued" || row.state === "needs_review")) {
|
|
118
130
|
// Read again each time: a queue entry's revision is the journal's, and every resolution moves it on.
|
|
119
131
|
bus.resolveDelivery(id, bus.queueShow(id)!.revision, "discard", "reset");
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { paint, quoted, wrap, type Tone } from "./console-state.ts";
|
|
1
|
+
import { paint, quoted, terminalText, wrap, type Tone } from "./console-state.ts";
|
|
2
2
|
import { operationLines, stepLabel } from "./upgrade-labels.ts";
|
|
3
3
|
import { recordedPeerGuidance, stopAndArchiveSummary, type RecordedPeerGuidance } from "./recovery-guidance.ts";
|
|
4
4
|
import { restoreReceiptStatus } from "./upgrade-restore-state.ts";
|
|
@@ -162,6 +162,16 @@ export function renderBlocked(op: RecoveryOperation, choices: readonly NextChoic
|
|
|
162
162
|
} else if (op.failure?.code === "database_busy") {
|
|
163
163
|
happened = "The recovery database could not complete this step yet.";
|
|
164
164
|
why = "The coordinator recorded database contention as retryable; its saved recovery state is retained.";
|
|
165
|
+
} else if (!cause && op.error) {
|
|
166
|
+
// A recorded error is evidence to show, not a typed classification. Reuse the console's
|
|
167
|
+
// terminal sanitizer before the normal renderer; discard later lines and bound by code point.
|
|
168
|
+
const first = terminalText(op.error.split(/[\r\n\p{Zl}\p{Zp}]/u, 1)[0] ?? "").replace(/\p{Cc}/gu, "").trim();
|
|
169
|
+
if (first) {
|
|
170
|
+
const points = Array.from(first);
|
|
171
|
+
const recorded = points.length > 512 ? `${points.slice(0, 509).join("")}...` : first;
|
|
172
|
+
why = `recorded error: ${recorded} (typed cause unknown).`;
|
|
173
|
+
action = "Resolve the recorded error, then resume or choose an offered action. If the target lacks a feature the source needs, choose a compatible target; keep the source records. Read [j] or use --verbose for details.";
|
|
174
|
+
}
|
|
165
175
|
}
|
|
166
176
|
const output = [...lines(`What happened: ${happened}`, "failure", options), ...lines(`Why: ${why}`, "muted", options), ...lines(`What to do: ${action}`, "attention", options)];
|
|
167
177
|
for (const choice of choices) {
|
|
@@ -19,6 +19,7 @@ import { abandonRestartSnapshot, readRestartSnapshot, waiveRecoveryPeers } from
|
|
|
19
19
|
import { readRecoveryContinuity, retireRecoveryContinuity, writeRecoveryContinuity } from "../hub/recovery-continuity.ts";
|
|
20
20
|
import { recoveryLock } from "../hub/recovery-store.ts";
|
|
21
21
|
import { decodeRestoreReceipt, restoreTransition, type RestoreReceipt, type RestoreEvidence, type RestoreTransition } from "./upgrade-restore-state.ts";
|
|
22
|
+
import { RecoveryDeferredHubError } from "../hub/deferred-hub.ts";
|
|
22
23
|
import { RecoveryPreservationError, RecoveryPresentationError, type RecoveryNativeCause } from "./upgrade-presentation-data.ts";
|
|
23
24
|
|
|
24
25
|
/** An unmanaged Claude's plugin retries with a backoff of at most 30 s; three of those bound the reconnect wait (#206). */
|
|
@@ -172,12 +173,21 @@ export async function inspectRecovery(project: Project, targetFeatures?: readonl
|
|
|
172
173
|
const sourceProtocol = control?.protocol;
|
|
173
174
|
const legacySupported = sourceProtocol !== undefined && RECOVERY_SOURCE_PROTOCOLS.includes(sourceProtocol as (typeof RECOVERY_SOURCE_PROTOCOLS)[number]);
|
|
174
175
|
// A stopped runtime reports whose unreleased restart snapshot it holds: only that one can start it again (#215).
|
|
175
|
-
|
|
176
|
+
let snapshot: string | undefined, snapshotError: Inspection["snapshotError"];
|
|
177
|
+
if (base.state === "stopped") {
|
|
178
|
+
try { snapshot = readRestartSnapshot(project.stateDir, { projectRoot: project.root, projectId: project.id })?.operationId; }
|
|
179
|
+
catch (error) {
|
|
180
|
+
if (!(error instanceof RecoveryDeferredHubError)) throw error;
|
|
181
|
+
// Corruption blocks restore, but cannot erase independently confirmed shutdown or the person's archive exit.
|
|
182
|
+
// Keep diagnostics static: the rejected snapshot may contain forged envelope fields.
|
|
183
|
+
snapshotError = { code: "recovery_deferred_hub", message: "restart snapshot deferred hub notices are invalid; restore is blocked" };
|
|
184
|
+
}
|
|
185
|
+
}
|
|
176
186
|
// A dead pid or a missing project directory is final whatever protocol its manifest names: probing it would read
|
|
177
187
|
// "unavailable" forever and wedge abort, resume and stop-and-archive (#215).
|
|
178
188
|
if (base.state === "stopped" || base.state === "missing" || (base.state !== "running" && !legacySupported)) return {
|
|
179
|
-
state: base.state, peers: [], blockers: base.state === "stopped" ? [] : [base.state === "incompatible" ? "manual-bootstrap-required: source lacks the recovery contract; use its matching CLI" : base.error ?? base.state],
|
|
180
|
-
...(control?.instanceId ? { instanceId: control.instanceId } : {}), ...(control?.protocol ? { protocol: control.protocol } : {}), ...(snapshot ? { snapshot } : {}),
|
|
189
|
+
state: base.state, peers: [], blockers: base.state === "stopped" ? snapshotError ? [snapshotError.message] : [] : [base.state === "incompatible" ? "manual-bootstrap-required: source lacks the recovery contract; use its matching CLI" : base.error ?? base.state],
|
|
190
|
+
...(control?.instanceId ? { instanceId: control.instanceId } : {}), ...(control?.protocol ? { protocol: control.protocol } : {}), ...(snapshot ? { snapshot } : {}), ...(snapshotError ? { snapshotError } : {}),
|
|
181
191
|
};
|
|
182
192
|
let status = base.status;
|
|
183
193
|
if (!status && control && sourceProtocol === PROTOCOL) return {
|
|
@@ -666,8 +676,8 @@ export function makeRecoveryDriver(run: RunCommand = runCommand, liveTestPeer?:
|
|
|
666
676
|
throw new Error(`source preparation expired or changed while waiting for the terminal; no terminal was closed`);
|
|
667
677
|
}
|
|
668
678
|
// Refresh the same prepared lease immediately before the terminal effect. Preserve
|
|
669
|
-
// the original peer roster
|
|
670
|
-
await control(planned.project, "prepare", op.id, planned.source.instanceId
|
|
679
|
+
// the original peer roster and verified target features, including after a resume.
|
|
680
|
+
await control(planned.project, "prepare", op.id, planned.source.instanceId!, op.targetFeatures);
|
|
671
681
|
progress.terminals[key] = "pending"; save();
|
|
672
682
|
const result = await closeTerminal(binding, 0, terminalOptions(run));
|
|
673
683
|
if (result.manualRequired) throw new Error(`${binding.peer}: terminal close not verified; manual-required`);
|
package/src/cli/upgrade.ts
CHANGED
|
@@ -32,6 +32,8 @@ export interface Inspection {
|
|
|
32
32
|
blockers: string[];
|
|
33
33
|
/** #215: for a stopped runtime, the operation whose committed restart snapshot its state dir holds (unreleased). */
|
|
34
34
|
snapshot?: string;
|
|
35
|
+
/** The runtime may be positively stopped while its restart snapshot remains unreadable; never claim ownership. */
|
|
36
|
+
snapshotError?: { code: "recovery_deferred_hub"; message: string };
|
|
35
37
|
}
|
|
36
38
|
export interface PlannedProject {
|
|
37
39
|
project: Project;
|
package/src/hub/bus.ts
CHANGED
|
@@ -2,6 +2,7 @@ import { appendNote, DIGEST, HUB, keepNotes, MAX_HOP, mergePrefaces, newEnvelope
|
|
|
2
2
|
import type { PeerAdapter, DeliveryReceipt } from "./peers.ts";
|
|
3
3
|
import { DeliveryJournal, MAX_ENVELOPE_BODY, type JournalDelivery, type JournalDeliveryState } from "./delivery-journal.ts";
|
|
4
4
|
import { DELIVERY_RETRY_WARNING } from "./delivery-hints.ts";
|
|
5
|
+
import { MAX_DEFERRED_HUB, RecoveryDeferredHubError, validateDeferredHub } from "./deferred-hub.ts";
|
|
5
6
|
import { PreEffectToolRefusal } from "./tool-refusal.ts";
|
|
6
7
|
|
|
7
8
|
export type { DeliveryReceipt } from "./peers.ts";
|
|
@@ -64,6 +65,8 @@ export interface BusSnapshot {
|
|
|
64
65
|
attempts: Record<string, number>;
|
|
65
66
|
withdrawn: string[];
|
|
66
67
|
manualPaused?: string[];
|
|
68
|
+
/** Hub notices accepted during recovery, durable but outside the carried delivery queues. */
|
|
69
|
+
deferredHub?: Envelope[];
|
|
67
70
|
journal?: ReturnType<DeliveryJournal["snapshot"]>;
|
|
68
71
|
}
|
|
69
72
|
|
|
@@ -142,6 +145,12 @@ export class Bus {
|
|
|
142
145
|
/** Synchronous native failure observation, before queue callbacks can finish turn metrics. */
|
|
143
146
|
onDeliveryFailed?: (peer: PeerId) => void;
|
|
144
147
|
private recoveryHeld = false;
|
|
148
|
+
private deferHub = false;
|
|
149
|
+
private readonly deferredHub = new Map<string, Envelope>();
|
|
150
|
+
private flushingHub = false;
|
|
151
|
+
private replayingHub = false;
|
|
152
|
+
private deferredEvents: BusEvent[] = [];
|
|
153
|
+
private deferredNudges: (() => void)[] = [];
|
|
145
154
|
private steering = 0;
|
|
146
155
|
private condensing = 0;
|
|
147
156
|
private closed = false;
|
|
@@ -273,10 +282,61 @@ export class Bus {
|
|
|
273
282
|
}
|
|
274
283
|
|
|
275
284
|
/** Stop new deliveries while a coordinator takes a stable snapshot. Existing adapter turns are left alone. */
|
|
276
|
-
setRecoveryHold(held: boolean): void {
|
|
277
|
-
this.
|
|
278
|
-
if (
|
|
279
|
-
|
|
285
|
+
setRecoveryHold(held: boolean, deferHub = false): void {
|
|
286
|
+
if (this.replayingHub) throw new RecoveryDeferredHubError("recovery notice release is already in progress");
|
|
287
|
+
if (held) {
|
|
288
|
+
this.recoveryHeld = true;
|
|
289
|
+
this.deferHub = deferHub;
|
|
290
|
+
// An older target cannot carry this additive field: materialize before commit's expected integrity.
|
|
291
|
+
if (!deferHub) this.flushDeferredHub();
|
|
292
|
+
return;
|
|
293
|
+
}
|
|
294
|
+
// Persist each deferred -> queued transition while deliveries are still fenced. A failed checkpoint
|
|
295
|
+
// leaves the hold and the original pending head intact; a crash after it resumes from the journal.
|
|
296
|
+
this.flushDeferredHub();
|
|
297
|
+
this.recoveryHeld = false;
|
|
298
|
+
this.onQueues?.();
|
|
299
|
+
for (const id of this.peers.keys()) void this.drain(id);
|
|
300
|
+
}
|
|
301
|
+
|
|
302
|
+
private flushDeferredHub(): void {
|
|
303
|
+
if (this.flushingHub) return;
|
|
304
|
+
if (this.storageError) this.persist();
|
|
305
|
+
this.replayingHub = true;
|
|
306
|
+
try { while (this.deferredHub.size) {
|
|
307
|
+
const env = this.deferredHub.values().next().value!;
|
|
308
|
+
const beforeDeferred = [...this.deferredHub], beforeSeen = [...this.seen], beforeWithdrawn = [...this.withdrawn];
|
|
309
|
+
const affected = new Map(this.audience(env).map(peer => {
|
|
310
|
+
const queue = this.queues.get(peer);
|
|
311
|
+
return [peer, queue ? { queue, contents: [...queue] } : undefined] as const;
|
|
312
|
+
}));
|
|
313
|
+
const outcomes = this.pendingOutcomes.length;
|
|
314
|
+
this.flushingHub = true;
|
|
315
|
+
this.deferredEvents = []; this.deferredNudges = [];
|
|
316
|
+
try {
|
|
317
|
+
this.deferredHub.delete(env.id);
|
|
318
|
+
this.publishNow(env);
|
|
319
|
+
} catch (error) {
|
|
320
|
+
// Only publication state is transactional. A live peer's preface, pauses, attempts and
|
|
321
|
+
// recovery mode are independent state, not something a failed checkpoint may reload.
|
|
322
|
+
this.deferredHub.clear(); for (const [id, pending] of beforeDeferred) this.deferredHub.set(id, pending);
|
|
323
|
+
this.seen.clear(); for (const [id, seen] of beforeSeen) this.seen.set(id, seen);
|
|
324
|
+
this.withdrawn.clear(); for (const id of beforeWithdrawn) this.withdrawn.add(id);
|
|
325
|
+
for (const [peer, queue] of affected) {
|
|
326
|
+
if (queue) { queue.queue.splice(0, queue.queue.length, ...queue.contents); this.queues.set(peer, queue.queue); }
|
|
327
|
+
else this.queues.delete(peer);
|
|
328
|
+
}
|
|
329
|
+
this.pendingOutcomes.length = outcomes;
|
|
330
|
+
this.deferredEvents = []; this.deferredNudges = [];
|
|
331
|
+
throw new RecoveryDeferredHubError("deferred hub notice checkpoint failed; recovery remains held; restore delivery storage, then retry release or abort");
|
|
332
|
+
} finally { this.flushingHub = false; }
|
|
333
|
+
// Publication observers run only after the durable transition. Reentrant notices join the FIFO tail.
|
|
334
|
+
const events = this.deferredEvents; this.deferredEvents = [];
|
|
335
|
+
for (const event of events) this.emit(event);
|
|
336
|
+
const nudges = this.deferredNudges; this.deferredNudges = [];
|
|
337
|
+
for (const nudge of nudges) nudge();
|
|
338
|
+
this.onQueues?.();
|
|
339
|
+
} } finally { this.replayingHub = false; }
|
|
280
340
|
}
|
|
281
341
|
|
|
282
342
|
get isRecoveryHeld(): boolean {
|
|
@@ -296,7 +356,7 @@ export class Bus {
|
|
|
296
356
|
let delay = 1;
|
|
297
357
|
while (this.draining.size || this.steering || this.condensing) {
|
|
298
358
|
if (Date.now() >= deadline) {
|
|
299
|
-
this.
|
|
359
|
+
if (!heldBefore) this.setRecoveryHold(false);
|
|
300
360
|
throw new Error(`recovery fence timed out after ${timeoutMs} ms with ${this.draining.size + this.steering + this.condensing} deliveries still in flight; retry the operation`);
|
|
301
361
|
}
|
|
302
362
|
await Bun.sleep(delay);
|
|
@@ -314,6 +374,7 @@ export class Bus {
|
|
|
314
374
|
attempts: Object.fromEntries(this.attempts),
|
|
315
375
|
withdrawn: [...this.withdrawn],
|
|
316
376
|
manualPaused: [...this.manualPaused],
|
|
377
|
+
...(this.deferredHub.size ? { deferredHub: structuredClone([...this.deferredHub.values()]) } : {}),
|
|
317
378
|
};
|
|
318
379
|
if (includeJournal && this.journal) out.journal = this.journal.snapshot();
|
|
319
380
|
return out;
|
|
@@ -341,6 +402,10 @@ export class Bus {
|
|
|
341
402
|
}
|
|
342
403
|
|
|
343
404
|
private loadSnapshot(snapshot: BusSnapshot): void {
|
|
405
|
+
const deferred = validateDeferredHub(snapshot.deferredHub, undefined, new Set([...snapshot.seen.map(e => e.id), ...Object.values(snapshot.queues).flat().map(e => e.id)]));
|
|
406
|
+
if (deferred.length) { this.recoveryHeld = true; this.deferHub = true; }
|
|
407
|
+
this.deferredHub.clear();
|
|
408
|
+
for (const env of deferred) this.deferredHub.set(env.id, structuredClone(env));
|
|
344
409
|
this.queues.clear();
|
|
345
410
|
for (const [id, queue] of Object.entries(snapshot.queues ?? {})) this.queues.set(id, queue.map((e) => ({ ...e, ...(e.to ? { to: [...e.to] } : {}) })));
|
|
346
411
|
this.prefaces.clear();
|
|
@@ -532,7 +597,7 @@ export class Bus {
|
|
|
532
597
|
|
|
533
598
|
queuedBacklogIds(id: PeerId): Set<number> {
|
|
534
599
|
const ids = new Set<number>();
|
|
535
|
-
const retained = [...
|
|
600
|
+
const retained = [...this.pendingEnvelopes(id), ...(this.journal?.unresolved(id).filter(row => row.state !== "queued").flatMap(row => row.originals) ?? [])];
|
|
536
601
|
for (const env of retained) {
|
|
537
602
|
const backlog = env.refs?.backlog;
|
|
538
603
|
if (env.from !== HUB || env.kind !== "task" || !Array.isArray(backlog) || backlog.length > 64) continue;
|
|
@@ -602,8 +667,52 @@ export class Bus {
|
|
|
602
667
|
this.prefaces.set(id, !now ? { ...env, body: uniqueMemoryLines(env.body) } : now.id === env.id ? { ...now, body: uniqueMemoryLines(now.body) } : { ...env, body: keepNotes(mergePrefaces(env.body, now.body), NOTES_KEPT) });
|
|
603
668
|
}
|
|
604
669
|
|
|
605
|
-
/**
|
|
670
|
+
/** Stopped-runtime reset only: audit every deferred recipient without publishing or releasing work. */
|
|
671
|
+
discardDeferredHub(reason: string): string[] {
|
|
672
|
+
if (this.peers.size || this.flushingHub || this.replayingHub || !this.journal || this.storageError) throw new Error("deferred reset requires a stopped, readable delivery journal");
|
|
673
|
+
const pending = [...this.deferredHub];
|
|
674
|
+
const settled: string[] = [];
|
|
675
|
+
try {
|
|
676
|
+
this.journal.transaction(() => {
|
|
677
|
+
for (const [, env] of pending) for (const peer of new Set(env.to ?? [])) {
|
|
678
|
+
const id = `q:${peer}:${env.id}`;
|
|
679
|
+
this.journal!.createDelivery({ id, peer, state: "queued", createdAt: env.ts, originals: [env], out: [env] });
|
|
680
|
+
const row = this.journal!.get(id)!;
|
|
681
|
+
this.journal!.resolve(id, row.revision, "discard", reason, row.revision, "user");
|
|
682
|
+
settled.push(id);
|
|
683
|
+
}
|
|
684
|
+
this.deferredHub.clear();
|
|
685
|
+
this.persist();
|
|
686
|
+
});
|
|
687
|
+
} catch (error) {
|
|
688
|
+
this.deferredHub.clear(); for (const [id, env] of pending) this.deferredHub.set(id, env);
|
|
689
|
+
throw error;
|
|
690
|
+
}
|
|
691
|
+
return settled;
|
|
692
|
+
}
|
|
693
|
+
|
|
694
|
+
/** Accepted pending work, including notices withheld from the recovery integrity view. */
|
|
695
|
+
pendingEnvelopes(peer: PeerId): Envelope[] {
|
|
696
|
+
return [...(this.queues.get(peer) ?? []), ...[...this.deferredHub.values()].filter(e => e.to?.includes(peer))];
|
|
697
|
+
}
|
|
698
|
+
|
|
699
|
+
/** Returns the peers the envelope was accepted for, including durable deferred notices. */
|
|
606
700
|
publish(env: Envelope): PeerId[] {
|
|
701
|
+
if (this.storageError) throw new Error("delivery journal unavailable");
|
|
702
|
+
if (this.seen.has(env.id) || this.deferredHub.has(env.id)) return [];
|
|
703
|
+
if (this.recoveryHeld && (this.deferHub || this.replayingHub) && env.from === HUB && env.priority !== "fyi" && env.hop <= MAX_HOP) {
|
|
704
|
+
if (this.deferredHub.size >= MAX_DEFERRED_HUB) throw new RecoveryDeferredHubError();
|
|
705
|
+
const targets = this.audience(env);
|
|
706
|
+
const copy = structuredClone({ ...env, to: targets });
|
|
707
|
+
this.deferredHub.set(copy.id, copy);
|
|
708
|
+
try { this.persist(); }
|
|
709
|
+
catch (error) { this.deferredHub.delete(copy.id); throw error; }
|
|
710
|
+
return targets;
|
|
711
|
+
}
|
|
712
|
+
return this.publishNow(env);
|
|
713
|
+
}
|
|
714
|
+
|
|
715
|
+
private publishNow(env: Envelope): PeerId[] {
|
|
607
716
|
if (this.storageError) throw new Error("delivery journal unavailable");
|
|
608
717
|
if (this.seen.has(env.id)) return [];
|
|
609
718
|
this.seen.set(env.id, env);
|
|
@@ -638,7 +747,7 @@ export class Bus {
|
|
|
638
747
|
} } finally { this.suppressDrain = false; }
|
|
639
748
|
if (this.journal) {
|
|
640
749
|
this.persist();
|
|
641
|
-
this.onQueues?.();
|
|
750
|
+
if (!this.flushingHub) this.onQueues?.();
|
|
642
751
|
for (const id of targets) {
|
|
643
752
|
const peer = this.peers.get(id);
|
|
644
753
|
if (!this.held(id) && env.priority === "important" && peer?.steer && this.stateOf(id) === "busy") void this.steerQueued(id, env);
|
|
@@ -681,7 +790,7 @@ export class Bus {
|
|
|
681
790
|
|
|
682
791
|
/** A recently published envelope, for resolving `reply_to`. */
|
|
683
792
|
get(id: string): Envelope | undefined {
|
|
684
|
-
return this.seen.get(id);
|
|
793
|
+
return this.seen.get(id) ?? this.deferredHub.get(id);
|
|
685
794
|
}
|
|
686
795
|
|
|
687
796
|
/** The envelopes most recently handed to a peer, used to distinguish a received request from an unread queue. */
|
|
@@ -696,10 +805,10 @@ export class Bus {
|
|
|
696
805
|
*/
|
|
697
806
|
withdraw(envelopeId: string): boolean {
|
|
698
807
|
this.markWithdrawn(envelopeId);
|
|
699
|
-
let removed =
|
|
808
|
+
let removed = this.deferredHub.delete(envelopeId);
|
|
700
809
|
for (const queue of this.queues.values()) {
|
|
701
810
|
const i = queue.findIndex((e) => e.id === envelopeId);
|
|
702
|
-
if (i !== -1)
|
|
811
|
+
if (i !== -1) { queue.splice(i, 1); removed = true; }
|
|
703
812
|
}
|
|
704
813
|
if (removed) this.onQueues?.();
|
|
705
814
|
this.persist();
|
|
@@ -709,8 +818,15 @@ export class Bus {
|
|
|
709
818
|
/** Revoke pending feed copies only; accepted/dispatching journal records and native turns are untouched. */
|
|
710
819
|
revokeSupervision(peer: PeerId): number {
|
|
711
820
|
const queue = this.queues.get(peer);
|
|
712
|
-
if (!queue && !this.preparing.has(peer)) return 0;
|
|
713
821
|
let removed = 0;
|
|
822
|
+
for (const [id, env] of this.deferredHub) {
|
|
823
|
+
if (!env.refs?.supervision || !env.to?.includes(peer)) continue;
|
|
824
|
+
const remaining = env.to.filter(target => target !== peer);
|
|
825
|
+
if (remaining.length) this.deferredHub.set(id, { ...env, to: remaining });
|
|
826
|
+
else this.deferredHub.delete(id);
|
|
827
|
+
removed++;
|
|
828
|
+
}
|
|
829
|
+
if (!queue && !this.preparing.has(peer)) { if (removed) this.persist(); return removed; }
|
|
714
830
|
const discarded = new Set<string>();
|
|
715
831
|
for (let i = (queue?.length ?? 0) - 1; i >= 0; i--) {
|
|
716
832
|
const env = queue![i]!;
|
|
@@ -796,9 +912,11 @@ export class Bus {
|
|
|
796
912
|
}
|
|
797
913
|
if (queuedEnv.priority === "important" && this.peers.get(id)?.pullOnly) {
|
|
798
914
|
// Advisory only (issue #398): the envelope stays queued for `pull`; the hook must never affect it.
|
|
799
|
-
|
|
915
|
+
const counts = { queued: queue.length, important: queue.filter((e) => e.priority === "important").length };
|
|
916
|
+
const nudge = () => { try { this.opts.onImportantQueued?.(id, counts); } catch { /* a nudge never stops an enqueue */ } };
|
|
917
|
+
if (this.flushingHub) this.deferredNudges.push(nudge); else nudge();
|
|
800
918
|
}
|
|
801
|
-
this.onQueues?.();
|
|
919
|
+
if (!this.flushingHub) this.onQueues?.();
|
|
802
920
|
if (!this.suppressDrain) this.persist();
|
|
803
921
|
if (!this.suppressDrain) void this.drain(id);
|
|
804
922
|
}
|
|
@@ -1060,6 +1178,7 @@ export class Bus {
|
|
|
1060
1178
|
}
|
|
1061
1179
|
|
|
1062
1180
|
private emit(e: BusEvent): void {
|
|
1181
|
+
if (this.flushingHub) { this.deferredEvents.push(e); return; }
|
|
1063
1182
|
for (const tap of this.taps) tap(e);
|
|
1064
1183
|
}
|
|
1065
1184
|
}
|
|
@@ -8,7 +8,7 @@ export function stateDirFor(cwd: string): string {
|
|
|
8
8
|
|
|
9
9
|
/** Control WS wire version. 2 = `deliver` carries `envs` (digests); 3 = `tools` role and task messages; 4 = budget messages and `hub_checkpoint`; 5 = `ask`; 6 = console-only `ui` session bootstrap; 8 = controlled recovery; 9 = Pi bridge metadata; 10 = durable delivery receipts; 11 = queue hold diagnostics; 12 = generation-bound channel settlement and execution budgets; 13 = turn-free facts (`facts` with tool, session and transcript binding, `silenced`) and per-recipient send results (issue #108). 14 = native context-window status and session-bound context checkpoint request ids (issue #185). 15 = conductor tools and permission deadlines/closed events (issues #190, #194). 16 = a Claude session without channel evidence: hello `channel: false` and `inbox` (issue #205). The plugin is installed apart from the daemon, so they can drift. */
|
|
10
10
|
export const PROTOCOL = 16;
|
|
11
|
-
export const DAEMON_FEATURES = ["task_cancel"] as const;
|
|
11
|
+
export const DAEMON_FEATURES = ["task_cancel", "recovery_deferred_hub"] as const;
|
|
12
12
|
/** Protocols a current coordinator may authenticate while upgrading a running source. */
|
|
13
13
|
export const RECOVERY_SOURCE_PROTOCOLS = [9, 10, 11, 12, 13, 14, 15, PROTOCOL] as const;
|
|
14
14
|
|
package/src/hub/daemon.ts
CHANGED
|
@@ -22,6 +22,7 @@ import { jcodeMcpReady } from "../cli/jcode-mcp.ts";
|
|
|
22
22
|
import { loadSdlc, sdlcGh } from "./sdlc.ts";
|
|
23
23
|
import { peerRosterConfig, rosterAllows, type PeerRoster } from "./roster.ts";
|
|
24
24
|
import { KNOWN_PEERS, OPTIONAL_PEERS, PLANNER_PEERS, SUPPORTED_PEERS, isSupportedPeer } from "./peer-catalog.ts";
|
|
25
|
+
import { RecoveryDeferredHubError } from "./deferred-hub.ts";
|
|
25
26
|
import { databaseBusySite, isDatabaseBusy, RecoveryDatabaseBusy } from "./recovery-error.ts";
|
|
26
27
|
import { buildKimiLaunch } from "../cli/launch.ts";
|
|
27
28
|
import { definitionOf } from "../adapters/registry.ts";
|
|
@@ -865,7 +866,7 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
865
866
|
let recoveryCommitted = false;
|
|
866
867
|
let recoveryPeerSnapshot: RestartPeerSnapshot[] | undefined;
|
|
867
868
|
let recoveryLeaseTimer: ReturnType<typeof setTimeout> | undefined;
|
|
868
|
-
const recoveryActive = () => !!recoveryOperationId && recoveryPhase !== "released";
|
|
869
|
+
const recoveryActive = () => !!recoveryOperationId && (recoveryPhase !== "released" || bus.isRecoveryHeld);
|
|
869
870
|
if (restored) {
|
|
870
871
|
bus.restore(restored.bus, restored.operationId);
|
|
871
872
|
for (const hold of conductorHolds.list()) bus.pause(hold.peer);
|
|
@@ -874,22 +875,25 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
874
875
|
recoveryOperationId = restored.operationId;
|
|
875
876
|
recoveryPhase = "restored";
|
|
876
877
|
recoveryPeerSnapshot = restored.peers;
|
|
877
|
-
bus.setRecoveryHold(true);
|
|
878
|
+
bus.setRecoveryHold(true, true);
|
|
879
|
+
} else {
|
|
880
|
+
// A prior release may have committed its queue checkpoint before this daemon was stopped.
|
|
881
|
+
bus.setRecoveryHold(false);
|
|
878
882
|
}
|
|
879
883
|
const recoveryPeerAllowed = (id: PeerId) => !recoveryActive() || !recoveryPeerSnapshot || recoveryPeerSnapshot.some((peer) => peer.id === id);
|
|
880
|
-
const armRecoveryLease = () => {
|
|
884
|
+
const armRecoveryLease = (delayMs = 10 * 60_000): void => {
|
|
881
885
|
clearTimeout(recoveryLeaseTimer);
|
|
882
886
|
recoveryLeaseTimer = setTimeout(() => {
|
|
883
887
|
if (!recoveryOperationId || recoveryCommitted || (recoveryPhase !== "preparing" && recoveryPhase !== "prepared")) return;
|
|
888
|
+
try { bus.setRecoveryHold(false); } catch (error) { log("recovery lease release deferred; delivery storage must be restored before abort"); armRecoveryLease(1000); return; }
|
|
884
889
|
log(`recovery lease expired for ${recoveryOperationId}; hold aborted`);
|
|
885
890
|
recoveryOperationId = undefined;
|
|
886
891
|
recoveryPhase = undefined;
|
|
887
892
|
recoveryPeerSnapshot = undefined;
|
|
888
|
-
bus.setRecoveryHold(false);
|
|
889
893
|
budget.setRecoveryHold(false);
|
|
890
894
|
removeRestartSnapshot(opts.stateDir);
|
|
891
895
|
writeStatus();
|
|
892
|
-
},
|
|
896
|
+
}, delayMs);
|
|
893
897
|
recoveryLeaseTimer.unref?.();
|
|
894
898
|
};
|
|
895
899
|
const memory = new MemoryClient(config.memory.worker_url ?? workerUrl(), 2000, log);
|
|
@@ -4465,7 +4469,7 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
4465
4469
|
armRecoveryLease();
|
|
4466
4470
|
recoveryPeerSnapshot ??= Object.values(recoveryPeers());
|
|
4467
4471
|
budget.setRecoveryHold(true);
|
|
4468
|
-
bus.setRecoveryHold(true);
|
|
4472
|
+
bus.setRecoveryHold(true, Array.isArray(msg.targetFeatures) && msg.targetFeatures.includes("recovery_deferred_hub"));
|
|
4469
4473
|
await bus.fenceRecovery();
|
|
4470
4474
|
{ const error = incompatibleCancellationTarget(); if (error) return recoveryError(error); }
|
|
4471
4475
|
const blocked = [...bus.peers].filter(([, peer]) => peer.state === "busy").map(([id]) => id);
|
|
@@ -4483,11 +4487,11 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
4483
4487
|
if (msg.op === "abort") {
|
|
4484
4488
|
if (recoveryCommitted) return recoveryError("a committed recovery cannot be aborted");
|
|
4485
4489
|
if (recoveryPhase === "released") return { t: "recovery", ok: true, recovery: recoveryView() };
|
|
4490
|
+
bus.setRecoveryHold(false);
|
|
4486
4491
|
recoveryOperationId = undefined;
|
|
4487
4492
|
recoveryPhase = undefined;
|
|
4488
4493
|
recoveryPeerSnapshot = undefined;
|
|
4489
4494
|
clearTimeout(recoveryLeaseTimer);
|
|
4490
|
-
bus.setRecoveryHold(false);
|
|
4491
4495
|
budget.setRecoveryHold(false);
|
|
4492
4496
|
writeStatus();
|
|
4493
4497
|
return { t: "recovery", ok: true, aborted: true, recovery: recoveryView() };
|
|
@@ -4495,6 +4499,7 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
4495
4499
|
if (msg.op === "commit") {
|
|
4496
4500
|
if ((recoveryPhase !== "prepared" && recoveryPhase !== "preparing") || !recoveryReady()) return recoveryError("recovery is not ready; inspect until peers are idle, approvals are complete and completion checks have finished");
|
|
4497
4501
|
recoveryPhase = "prepared";
|
|
4502
|
+
bus.setRecoveryHold(true, Array.isArray(msg.targetFeatures) && msg.targetFeatures.includes("recovery_deferred_hub"));
|
|
4498
4503
|
recoveryPeerSnapshot ??= Object.values(recoveryPeers());
|
|
4499
4504
|
const currentPeers = recoveryPeers();
|
|
4500
4505
|
const peers = (recoveryPeerSnapshot ?? Object.values(currentPeers)).map((saved) => ({
|
|
@@ -4525,7 +4530,10 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
4525
4530
|
return { t: "recovery", ok: true, committed: true, recovery: { ...recoveryView(), phase: "prepared" } };
|
|
4526
4531
|
}
|
|
4527
4532
|
if (msg.op === "release") {
|
|
4528
|
-
if (recoveryPhase === "released")
|
|
4533
|
+
if (recoveryPhase === "released") {
|
|
4534
|
+
bus.setRecoveryHold(false); budget.setRecoveryHold(false);
|
|
4535
|
+
return { t: "recovery", ok: true, released: true, recovery: recoveryView() };
|
|
4536
|
+
}
|
|
4529
4537
|
if (recoveryPhase !== "restored") return recoveryError("recovery is not restored");
|
|
4530
4538
|
await bus.fenceRecovery();
|
|
4531
4539
|
const current = recoveryPeers();
|
|
@@ -4797,7 +4805,8 @@ export async function startDaemon(opts: DaemonOptions) {
|
|
|
4797
4805
|
try { log(`recovery ${sanitize(String(msg.op))} database contention: ${databaseBusySite(error).join("\n")}`); } catch { /* the reply still settles its rid */ }
|
|
4798
4806
|
const busy = new RecoveryDatabaseBusy();
|
|
4799
4807
|
reply({ ...recoveryError(busy.message), code: busy.code, retryable: busy.retryable });
|
|
4800
|
-
} else reply(recoveryError(
|
|
4808
|
+
} else if (error instanceof RecoveryDeferredHubError) reply({ ...recoveryError(error.message), code: error.code });
|
|
4809
|
+
else reply(recoveryError((error as Error)?.message ?? "recovery operation failed"));
|
|
4801
4810
|
});
|
|
4802
4811
|
return;
|
|
4803
4812
|
case "send": {
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
import { HUB, MAX_HOP, type Envelope } from "./envelope.ts";
|
|
2
|
+
|
|
3
|
+
/** ponytail: cap recovery at 2048 unpublished hub originals; replace with a paged durable outbox if measured recovery workloads exceed it. */
|
|
4
|
+
export const MAX_DEFERRED_HUB = 2048;
|
|
5
|
+
|
|
6
|
+
/** Refuses admission without dropping already accepted deferred work. */
|
|
7
|
+
export class RecoveryDeferredHubError extends Error {
|
|
8
|
+
readonly code = "recovery_deferred_hub";
|
|
9
|
+
constructor(message = "recovery deferred hub notice limit reached; release or abort recovery before publishing more notices") {
|
|
10
|
+
super(message);
|
|
11
|
+
this.name = "RecoveryDeferredHubError";
|
|
12
|
+
}
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
/** Journal-less restore uses the same bounded core envelope shape as durable deferred work. */
|
|
16
|
+
export function validateDeferredEnvelope(value: unknown, name: string): Envelope {
|
|
17
|
+
if (!value || typeof value !== "object" || Array.isArray(value))
|
|
18
|
+
throw new RecoveryDeferredHubError(`${name} must be an envelope`);
|
|
19
|
+
const env = value as Record<string, unknown>;
|
|
20
|
+
const text = (value: unknown, max: number) => typeof value === "string" && value.length > 0 && value.length <= max;
|
|
21
|
+
if (!text(env.id, 256) || !text(env.trace, 256) || env.from !== HUB ||
|
|
22
|
+
typeof env.body !== "string" || env.body.length > 1_000_000 ||
|
|
23
|
+
!Number.isSafeInteger(env.hop) || (env.hop as number) < 0 || (env.hop as number) > MAX_HOP ||
|
|
24
|
+
!Number.isSafeInteger(env.ts) || (env.ts as number) < 0 ||
|
|
25
|
+
!["chat", "task", "review", "status", "budget", "presence"].includes(env.kind as string) ||
|
|
26
|
+
!["important", "status", "fyi"].includes(env.priority as string) ||
|
|
27
|
+
(!Array.isArray(env.to) || env.to.length > 256 || !env.to.every(peer => text(peer, 256))) ||
|
|
28
|
+
(env.private !== undefined && typeof env.private !== "boolean"))
|
|
29
|
+
throw new RecoveryDeferredHubError(`${name} is invalid`);
|
|
30
|
+
if (env.refs !== undefined) {
|
|
31
|
+
if (!env.refs || typeof env.refs !== "object" || Array.isArray(env.refs)) throw new RecoveryDeferredHubError(`${name}.refs is invalid`);
|
|
32
|
+
const refs = env.refs as Record<string, unknown>;
|
|
33
|
+
for (const key of ["repo", "branch", "commit", "task"])
|
|
34
|
+
if (refs[key] !== undefined && !text(refs[key], 4096)) throw new RecoveryDeferredHubError(`${name}.refs.${key} is invalid`);
|
|
35
|
+
if (refs.paths !== undefined && (!Array.isArray(refs.paths) || refs.paths.length > 256 || !refs.paths.every(path => text(path, 4096))))
|
|
36
|
+
throw new RecoveryDeferredHubError(`${name}.refs.paths is invalid`);
|
|
37
|
+
}
|
|
38
|
+
return value as Envelope;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
/** Callers keep their existing envelope validation; this leaf checks deferred-only invariants. */
|
|
42
|
+
export function validateDeferredHub(
|
|
43
|
+
value: unknown,
|
|
44
|
+
envelope: (value: unknown, name: string) => Envelope = validateDeferredEnvelope,
|
|
45
|
+
occupied: ReadonlySet<string> = new Set(),
|
|
46
|
+
name = "bus.deferredHub",
|
|
47
|
+
): Envelope[] {
|
|
48
|
+
if (value === undefined) return [];
|
|
49
|
+
if (!Array.isArray(value) || value.length > MAX_DEFERRED_HUB)
|
|
50
|
+
throw new RecoveryDeferredHubError(`${name} must be an array of at most ${MAX_DEFERRED_HUB} hub envelopes`);
|
|
51
|
+
const ids = new Set<string>();
|
|
52
|
+
return value.map((item, index) => {
|
|
53
|
+
validateDeferredEnvelope(item, `${name}[${index}]`);
|
|
54
|
+
const env = envelope(item, `${name}[${index}]`);
|
|
55
|
+
if (env.from !== HUB) throw new RecoveryDeferredHubError(`${name}[${index}] is not hub-originated`);
|
|
56
|
+
if (ids.has(env.id) || occupied.has(env.id)) throw new RecoveryDeferredHubError(`${name}[${index}] repeats an accepted envelope id`);
|
|
57
|
+
ids.add(env.id);
|
|
58
|
+
return env;
|
|
59
|
+
});
|
|
60
|
+
}
|
|
@@ -6,6 +6,7 @@ import { HUB, USER, type Envelope, type Kind } from "./envelope.ts";
|
|
|
6
6
|
import { heldDeliveryHints } from "./delivery-hints.ts";
|
|
7
7
|
import { processLiveness } from "../pi/process-signature.ts";
|
|
8
8
|
import { HUB_DB_BUSY_TIMEOUT_MS } from "./db-timeout.ts";
|
|
9
|
+
import { validateDeferredHub } from "./deferred-hub.ts";
|
|
9
10
|
|
|
10
11
|
export type JournalDeliveryState =
|
|
11
12
|
| "queued"
|
|
@@ -167,6 +168,11 @@ function validateBus(value: unknown, name = "bus"): BusSnapshot {
|
|
|
167
168
|
list(b.manualPaused, `${name}.manualPaused`, 4096).forEach((id) =>
|
|
168
169
|
text(id, `${name}.manualPaused[]`, 256),
|
|
169
170
|
);
|
|
171
|
+
const occupied = new Set<string>([
|
|
172
|
+
...(b.seen as Envelope[]).map(env => env.id),
|
|
173
|
+
...Object.values(queues).flatMap(items => (items as Envelope[]).map(env => env.id)),
|
|
174
|
+
]);
|
|
175
|
+
validateDeferredHub(b.deferredHub, validateEnvelope, occupied, `${name}.deferredHub`);
|
|
170
176
|
if (b.journal !== undefined) fail("nested journal snapshot is invalid");
|
|
171
177
|
return b as unknown as BusSnapshot;
|
|
172
178
|
}
|
package/src/hub/restart.ts
CHANGED
|
@@ -2,10 +2,11 @@ import { chmodSync, mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSy
|
|
|
2
2
|
import { createHash } from "node:crypto";
|
|
3
3
|
import { join } from "node:path";
|
|
4
4
|
import type { BusSnapshot } from "./bus.ts";
|
|
5
|
-
import type { PeerId, PeerState } from "./envelope.ts";
|
|
5
|
+
import type { Envelope, PeerId, PeerState } from "./envelope.ts";
|
|
6
6
|
import { MAX_HOP } from "./envelope.ts";
|
|
7
7
|
import { validateJournalSnapshot } from "./delivery-journal.ts";
|
|
8
8
|
import { atomicPrivateJSON } from "./recovery-store.ts";
|
|
9
|
+
import { RecoveryDeferredHubError, validateDeferredHub } from "./deferred-hub.ts";
|
|
9
10
|
|
|
10
11
|
export const RESTART_SCHEMA_VERSION = 1;
|
|
11
12
|
|
|
@@ -75,6 +76,16 @@ export function readRestartSnapshot(stateDir: string, expected: RestartReadExpec
|
|
|
75
76
|
const bus = s.bus as Partial<BusSnapshot>;
|
|
76
77
|
if (bus.schemaVersion !== 1 || !bus.queues || typeof bus.queues !== "object" || !bus.prefaces || typeof bus.prefaces !== "object" || !Array.isArray(bus.seen) || !bus.attempts || typeof bus.attempts !== "object" || !Array.isArray(bus.withdrawn)) return undefined;
|
|
77
78
|
if (!Object.keys(bus.queues).every(peerId) || !Object.keys(bus.prefaces).every(peerId) || !Object.values(bus.queues).every((queue) => Array.isArray(queue) && queue.every((e) => isEnvelope(e))) || !Object.values(bus.prefaces).every((e) => isEnvelope(e)) || !bus.seen.every((e) => isEnvelope(e, true)) || !bus.withdrawn.every((id) => typeof id === "string" && id.length > 0) || !Object.values(bus.attempts).every((n) => Number.isSafeInteger(n) && n > 0)) return undefined;
|
|
79
|
+
try {
|
|
80
|
+
const occupied = new Set<string>([
|
|
81
|
+
...bus.seen.map(env => env.id),
|
|
82
|
+
...Object.values(bus.queues).flatMap(queue => queue.map(env => env.id)),
|
|
83
|
+
]);
|
|
84
|
+
validateDeferredHub(bus.deferredHub, (value, name) => {
|
|
85
|
+
if (!isEnvelope(value)) throw new Error(`${name} is invalid`);
|
|
86
|
+
return value as Envelope;
|
|
87
|
+
}, occupied);
|
|
88
|
+
} catch (error) { if (error instanceof RecoveryDeferredHubError) throw error; return undefined; }
|
|
78
89
|
if (!Array.isArray(s.manualPaused) || !s.manualPaused.every((id) => typeof id === "string") || !Array.isArray(s.peers) || !s.peers.every((peer) => peer && typeof peer === "object" && typeof peer.id === "string" && ["idle", "busy", "paused", "offline"].includes(peer.state as string) && Array.isArray(peer.queueIds) && peer.queueIds.every((id) => typeof id === "string"))) return undefined;
|
|
79
90
|
if (s.integrity !== undefined && (!s.integrity || typeof s.integrity !== "object" || typeof s.integrity.boardDigest !== "string" || typeof s.integrity.budgetDigest !== "string" || !Array.isArray(s.integrity.manualPaused) || !s.integrity.manualPaused.every((id) => typeof id === "string") || !s.integrity.queues || typeof s.integrity.queues !== "object" || !Object.values(s.integrity.queues).every((ids) => Array.isArray(ids) && ids.every((id) => typeof id === "string")))) return undefined;
|
|
80
91
|
if (bus.journal !== undefined) { try { validateJournalSnapshot(bus.journal); } catch { return undefined; } }
|
package/src/hub/tasks.ts
CHANGED
|
@@ -2415,7 +2415,7 @@ export class Tasks {
|
|
|
2415
2415
|
const acceptance = previous.history.slice(assignment + 1).some(entry => entry.event === "accepted" && entry.by === peer);
|
|
2416
2416
|
const boardEvidence = role === "owner" && acceptance;
|
|
2417
2417
|
const mayHaveReceived = boardEvidence || this.d.bus.lastDeliveredTo(peer).some(matches) || this.d.bus.queueList(peer).some(row => ["dispatching", "accepted", "needs_review", "completed"].includes(row.state) && row.originals.some(matches));
|
|
2418
|
-
for (const env of this.d.bus.
|
|
2418
|
+
for (const env of this.d.bus.pendingEnvelopes(peer)) if (matches(env)) this.d.bus.withdraw(env.id);
|
|
2419
2419
|
if (!mayHaveReceived) return;
|
|
2420
2420
|
const body = role === "owner"
|
|
2421
2421
|
? `Task ${this.publicTitle(next)} is no longer assigned to you${next.owner ? `; its owner is now ${next.owner}` : "; it is unassigned"}. Do not accept or complete it; late calls under your former assignment are refused.`
|