@staix/agent-hub 0.12.3 → 0.12.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -0
- package/docs/cooperbench.md +35 -2
- package/docs/events.md +9 -0
- package/docs/operations.md +173 -5
- package/docs/specs/2026-09-19-agent-hub-design.md +31 -1
- package/docs/verification/2026-10-02-0.12.4.md +82 -0
- package/docs/verification/2026-10-03-0.12.5.md +46 -0
- package/package.json +1 -1
- package/plugins/agent-hub/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-hub/server.js +5 -4
- package/src/adapters/claude-channel.ts +2 -1
- package/src/adapters/codex-appserver.ts +51 -6
- package/src/cli/facts-hook.ts +40 -0
- package/src/cli/launch.ts +27 -3
- package/src/cli/main.ts +31 -2
- package/src/cli/upgrade-runtime.ts +1 -1
- package/src/hub/board.ts +4 -2
- package/src/hub/bus.ts +75 -3
- package/src/hub/child-process.ts +223 -2
- package/src/hub/cohorts.ts +308 -0
- package/src/hub/control-client.ts +3 -3
- package/src/hub/daemon.ts +389 -15
- package/src/hub/events.ts +9 -0
- package/src/hub/facts.ts +853 -0
- package/src/hub/report.ts +3 -0
- package/src/hub/routing.ts +94 -0
- package/src/hub/tasks.ts +412 -20
- package/templates/AGENTS.block.md +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,21 @@
|
|
|
2
2
|
|
|
3
3
|
Issue and pull request numbers in the entries for 0.7.7 and earlier refer to the previous repository, archived on 2026-09-30 when this repository's history was rewritten; the one exception is the open smoke-check issue, formerly #12, which moved here as #1. Numbers in newer entries refer to this repository.
|
|
4
4
|
|
|
5
|
+
## 0.12.5
|
|
6
|
+
|
|
7
|
+
- The Codex app-server runs in its own process group and is stopped as one, with what it started in groups of its own (MCP servers, tool commands): after SIGTERM, to its group and to each recorded process that leads a group of its own, and a grace period in which what it starts is recorded, the tree is frozen, read again and killed, and the stop is done only when the table shows none of it (or, when no table can be read at all, when its own group is gone); a launcher that exited before the stop while its group still has members fails the stop, its group unsignalled. `codex` is a node launcher, and mid-turn the native app-server does not exit on SIGTERM: the SIGKILL that followed reached the launcher alone, leaving the app-server at work under init and the hub process alive after `ahub kill` reported it stopped. The CooperBench runner's teardown is verified (#113): every process an arm starts that the reads see is recorded with its pid, start time and the evidence that it is the arm's (a fixture name in an argv is never proof on its own), read again every 5 s while the agents work and while Claude ends its turn (one that detaches between two reads, outside the fixture and without it in its argv, is not seen); a process with a fixture in its argv or as its working directory that is not proved the arm's is never signalled and keeps the cleanup open; teardown pauses the agents, lets a completed arm's Claude end its turn (up to 30 s, by the transcript's `turn_duration` row proved on the probe turn), asks for the normal shutdown, reads the process table back in the C locale and in UTC (a start time is part of a process's identity), signals only re-read identities, and records `clean`, `clean_with_fallback` or `incomplete_or_unknown` apart from the end reason. Evidence is taken after it, and the read locks and protected inputs come off only after a complete cleanup (otherwise `scripts/benchmarks/restore.ts` does it later, once the runner, every recorded process and anything in the fixture are gone). Run records carry the completion, cleanup, restoration, stage times and a summary of the Codex skills `skills/list` reports (its answer itself is not kept); a record from before 0.12.5 is judged by its own cleanup and trust flags, as then, and its teardown is shown as not verified (0.12.3 and 0.12.4 recorded only that the shutdown steps reported success, with no process readback); `teardown.ts` is pinned with the runner sources; runner commands run in their own process groups.
|
|
8
|
+
- Turn-free facts: files seen under a named directory beyond the 200 followed have a notice of their own; it and the touched-limit notice are spent only when an offer carrying them is read back (a name a PII pattern matches is counted, never named; an offer read back after its file was covered again and dropped again does not spend the later notice), and a new session hears only the drops of its own; it also starts its touched list afresh. A directory file rewritten with HEAD's bytes, which git lists until it refreshes its index, is neither named nor counted against `current()`; a file too large to read is compared by git's blob id of its raw bytes while all such files in one comparison total 64 MiB or less, and named otherwise (git's filters are not applied, so an end-of-line conversion or LFS makes it read as changed). The `fact` event carries `named` (directory files named without a diff), and the ledger counts them (#112).
|
|
9
|
+
- Split predictions (shadow only, routing unchanged): every hand-over records the new owner's profile (hub version, agent version from app-server or Claude Code's transcript, coordination mode); the other owner of an overlapping task not started yet counts as available while it is busy taking it (an owner goes busy as its task is delivered); observations accumulate across hub runs, one per hand-over (a decline or an escalation away counts against the peer that failed), and count only with the peer's current profile. A prediction is recorded when routing chooses the first owner of a task overlapping another owner's task not started yet (`where: "routing"`, the calibration record; `route explain` shows the same pair); cohort-time records carry `where: "cohort"` (#109).
|
|
10
|
+
- CooperBench manifest v2 and the #106 ablation pin hub 0.12.5, Codex 0.160.0 and Claude Code 2.1.288.
|
|
11
|
+
|
|
12
|
+
## 0.12.4
|
|
13
|
+
|
|
14
|
+
- A completed-change or edit-conflict notice is checked again for each recipient right before it is handed over, after condensation: a copy whose task has closed, changed owner or is gone is dropped instead of starting a turn, recorded as discarded and as a `stale` event. Other recipients and unrecorded envelopes are delivered as before (#106).
|
|
15
|
+
- Opt-in `coordination: "turn-free"` (advisory stays the default): owners of overlapping tasks form a cohort, silent only when every owner's context path is verified and no PII task is open. Messages between members are held back for those members only, with an explicit sender result and no charge against the sender's limits when nobody got them, until the sender has stopped after its task closed; that settlement is never undone, so a member's later turns are new work. A PII task opening lifts silent cohorts for good and says so. The member that finishes last (a console done counts as an intent) is asked to integrate, and its next done counts only for the same owner, cohort revision and files once the others have stopped; after three requests the outcome is recorded as unresolved. After a restart each peer's open overlapping tasks are told that overlaps are settled by message again, with the completed-change notices the silence held (#107).
|
|
16
|
+
- Turn-free facts at tool boundaries: what changed in an owner's files since it acknowledged them, with attribution only on effect evidence, offered through Claude's hooks (`ahub claude` adds them before and after every tool call and at Stop) or by steer into a running Codex turn, and acknowledged by a readback in the native session (facts sent with an integration request: by the next done). A Claude Read counts as seen only when it returned the whole file; a diff that matches a PII pattern is not shown. Contained paths, small regular files, tracking reset after PII. Control protocol 13; 0.12.3 (protocol 12) remains an upgrade source (#108).
|
|
17
|
+
- The split rule is a shadow prediction only: routing never changes; an overlap that forms or changes a cohort records a `split` event for the task, routed or named, and `ahub route explain` shows the trace, unknown unless its assumptions hold (an overlapping task its owner has started is other work) (#109).
|
|
18
|
+
- CooperBench manifest v2 adds a turn-free arm with hooks and status lines equal across arms and a planned-attempt check; a separate manifest is the #106 ablation (`experiments.stale_notices: "deliver"`); the grader grades completed and timed-out attempts and refuses a turn-free attempt whose tasks did not stay in one silent cohort, or any attempt whose records show a hook that is not the hub's; repeats rotate the arm order; run records carry their hook and MCP conditions; `scripts/benchmarks/ledger.py` reports every measure with its unit and coverage over one or several run directories, including in-task and whole-attempt usage per agent with tokens, completion and native settlement times, facts, hook timings and held-back messages, the grader's validity gates, and identifier and fragment contributions (#110).
|
|
19
|
+
|
|
5
20
|
## 0.12.3
|
|
6
21
|
|
|
7
22
|
- Fix live Claude delivery settlement holds, add generation-bound explicit completion and pending-settlement status (#100).
|
package/docs/cooperbench.md
CHANGED
|
@@ -28,7 +28,7 @@ The manifest's archive and prompt hashes are checked before a fixture is accepte
|
|
|
28
28
|
|
|
29
29
|
Stage `case-00.json` through `case-09.json` outside the repository. Each JSON file supplies the two exact feature prompts in feature order and the official evaluator's private case data. Pass each prior native Claude/Codex transcript as a separate `--protect <file>` argument. Do not expose those files, upstream evaluator/data tree, or prior run artifacts to the agents.
|
|
30
30
|
|
|
31
|
-
Prepare a new private run directory, then execute a fixed cohort. `--cases 0` runs
|
|
31
|
+
Prepare a new private run directory, then execute a fixed cohort. `--cases 0` runs every predeclared arm of the manifest for case zero (three in v1, four in v2). It never selects individual arms or retries only a scored subset.
|
|
32
32
|
|
|
33
33
|
```sh
|
|
34
34
|
python3 scripts/benchmarks/runner.py prepare \
|
|
@@ -51,7 +51,40 @@ bun scripts/benchmarks/native.ts \
|
|
|
51
51
|
|
|
52
52
|
The runner registers each exact fixture root with Orca, uses the project CLI to start and stop each daemon, and requires exact worktree/cwd readback for native sessions. Claude starts through the canonical `ahub claude` guard and loads this checkout's candidate bundle through an exact session-only `--mcp-config` server (`server:agent-hub`); it does not promote or mutate the globally installed plugin. Codex uses the native app-server adapter and `workspace-write` sandbox. Each agent must execute a setup-only `head -c 1` probe against the exact protected file and produce only the denied marker before scored tasks begin. There is one native sandbox layer per agent.
|
|
53
53
|
|
|
54
|
-
Every run record, diff, PTY/provider trace and evaluation artifact stays in the mode-0700 private run directory. Setup errors, structured provider quota errors, hub budget pauses, unsettled deliveries and interruptions stay separate and unscored. A quota snapshot or numeric `429` alone is not a provider error.
|
|
54
|
+
Every run record, diff, PTY/provider trace and evaluation artifact stays in the mode-0700 private run directory. Setup errors, structured provider quota errors, hub budget pauses, unsettled deliveries and interruptions stay separate and unscored. A quota snapshot or numeric `429` alone is not a provider error. On exit the runner restores the exact input and read-lock modes, the sibling artifact modes and the scoped Claude trust flag (removing its own fresh project entry), under the condition below. Teardown (issue #113) proves what it stops. Each process an arm starts is recorded with its pid and start time and the evidence that it is the arm's: the daemon by the pid in its state directory and an argv that serves this fixture (a later daemon there is a replacement, never adopted), Claude's launch chain by the arm's own session id as its `--session-id` (and the Orca terminal shell that runs it, working in the fixture), the Codex app-server as the daemon's child. What runs below them, and what is in a group a recorded process leads while that group is known to be the same one (its leader alive, or a recorded member still in it), is the arm's too; the table is read every 5 s while the agents work and during the completion wait, so a tool command's background job is recorded while its parent runs (one that detaches between two reads, outside the fixture and without it in its argv, is not seen). A fixture name in an argv is never proof on its own (Codex's need not name the fixture), and every recorded actor counts even when a read of the table fails. Teardown pauses every agent, reads the deliveries still owed, lets a completed arm's Claude end the turn it is in (up to 30 s and never past the 300 s limit (the bound is taken when the wait starts; one 250 ms poll can pass it), judged by the `turn_duration` row Claude Code writes after a turn, proved on the arm's probe turn; a stopped or timed-out arm does not wait; a completed arm's tree is hashed at the end of its active time and again after the teardown, and a write in between flags the attempt invalid), asks for the normal shutdown (Claude's terminal closed, `ahub kill`, the registration removed; a step of it that fails is recorded in `cleanup.normal.errors` and does not by itself fail the attempt, since the process readback decides, so a registration left behind is listed there), and reads the process table back, in the C locale and in UTC (start times are read as identities, the same in every reader). What is still running after a settle period gets SIGTERM; what is left then is frozen (SIGSTOP), the table read again, and it is killed (SIGKILL), what was frozen and not killed being continued: a frozen process starts nothing, so that read sees all of it. Every signal goes only to an identity read again just before, by the group the table shows it in now; a group leader is signalled with its group. What the read after the freeze shows for the first time is frozen too and read again before anything is killed, and when such a read fails only what a STOP reached is killed, by the last read that showed it (a stopped process keeps its pid). Anything with the fixture in its argv or as its working directory that is not proved the arm's (a background job that escaped the reads, someone's shell) is never signalled and keeps the cleanup open; the record keeps its pid, start time, program name and working directory, never its arguments. The record says `clean`, `clean_with_fallback` or `incomplete_or_unknown` (`cleanup`, with the normal shutdown's errors, every signal sent, what is still running and what is unresolved), and `cleanup_complete` is true only for the first two. Evidence follows the teardown: the transcript prefix (`transcriptBytes`, `transcriptSha256`, with the session and time it was taken) and the patch. The sibling read locks come off and the protected inputs become readable again only when the cleanup is complete; otherwise both stay, the record goes to `recovery/` beside the run's restoration ledger (which lists every arm's recorded processes and the runner's own identity), the cohort stops, and `bun scripts/benchmarks/restore.ts --run <run>` (also `runner.py restore`, which calls it) puts the modes back and takes back the trust entry once the runner, every recorded process and anything with a fixture in its argv or as its working directory are gone, then moves the kept records into `runs/`, where grading reads them as unavailable. A ledger written before runner identities needs `--runner-exited`, the operator's statement that the runner is gone. Runner commands are bounded (a hung one is killed), and SIGHUP stops the run like SIGINT and SIGTERM. `restoration.json` says `restored: true` only when the protected inputs and every arm's sibling locks are back. The trust entry is taken back with them (by the recovery when the cleanup is incomplete), and kept when the user changed it meanwhile. Orca currently has no repo removal command, so inactive exact-path fixture repos remain registered after a run; one case leaves one record per arm. Run records carry `completion`, `cleanup`, `restoration` and `stages` (completion wait, shutdown, settle, fallback with the final readback, evidence and restoration times), apart from the end reason: `end_reason_detail` keeps the original classification, and `end_flags` (an unverified model, modified metadata, a tree changed or unverified after the active time) makes any end other than an interruption, a provider quota error or a budget pause an infrastructure error beside it, which grading and the ledger report as `end_story`; a record with `teardown_errors` (a patch, transcript prefix, last capture, fixture metadata or event log that could not be taken, or sibling read locks or the trust entry that could not be put back, a concurrent change of the trust entry included), an incomplete cleanup or a trust entry not taken back is unavailable to grading and to the ledger alike (a record from before 0.12.5 is judged as it was then, by its own `cleanup_complete` and `trust_restored`, and the ledger shows its teardown with `verified: false`: 0.12.3 and 0.12.4 set `cleanup_complete` when the shutdown steps reported success (commands, terminal close, lock and trust restores), with no process readback, which is no proof that the processes were gone); runner commands, the `ps` reads included, run in process groups of their own, so a Ctrl-C reaches the runner, which stops in this order, and not the command it is running; the completion wait's tree hash never writes the agent's index. `teardown.ts` and the process table it reads (`src/hub/child-process.ts`) are pinned in `prepared.json`, and `teardown.ts` in `cohort.json` and `grade.json` too, with the other runner sources.
|
|
55
|
+
|
|
56
|
+
## Manifest v2: the turn-free arm
|
|
57
|
+
|
|
58
|
+
`scripts/benchmarks/manifest-v2.json` keeps v1's cases, models and limits and adds a fourth arm, `hub-turnfree-codex-claude` (issue #110): the same two agents and assignment rotation as `hub-codex-claude`, with `coordination: "turn-free"` in the fixture's hub config and fixture instructions that tell the owners not to message each other. A v1 manifest still validates and prepares. Its cohorts are run, graded and reported with the release and runner sources they were prepared with: `native.ts` refuses a manifest whose hub version is not the checkout's (v1 pins 0.12.3), and `runner.py` refuses to grade a cohort whose runner sources differ from its own.
|
|
59
|
+
|
|
60
|
+
Hooks are equal across arms: no arm runs the user's or a plugin's hooks, and no arm runs a status line (`disableAllHooks` turns it off in the other arms, so the turn-free arm leaves it out; Claude's quota reaches the hub in no arm). Claude starts with `--setting-sources project` and `--strict-mcp-config`; the solo and advisory arms set `disableAllHooks`, and the turn-free arm's session settings carry the hub's own hooks (before and after every tool call, and at Stop) and nothing else, because they are the treatment. Every Codex thread starts with `features.hooks` off: the turn-free arm's Codex boundary is the adapter's steer into the running turn, whose readback is the steered input coming back as a user message item. MCP servers are isolated too: Claude has only the hub's (`--strict-mcp-config`), and every Codex runs through a wrapper written into each arm's fixture that turns off the user's plugins, apps, sub-agents and turn-end notifier and disables each MCP server the user's config defines, by name, so it starts only the hub's; the user's config is not changed. Instructions: Claude reads the fixture's `AGENTS.md` through `--append-system-prompt-file` (Claude Code reads `CLAUDE.md`, not `AGENTS.md`), Codex as the project's `AGENTS.md`; Codex also reads the user's global `AGENTS.md`, which only a separate Codex home with its own login would leave out (the run records say so), and the user's Codex skills stay available to it (the wrapper leaves them on; `conditions.codex.skills` records what app-server's `skills/list` reports for the fixture: counts by scope, how many are enabled, and a hash of their names, never their bodies or paths, and the `skills/list` answer itself is not kept in `codexMessages`; Claude's Skill tool is denied); both are the same in every arm. Each run record carries these `conditions`; its `events` carry the hub's `capability` readbacks.
|
|
61
|
+
|
|
62
|
+
Validity, decided by the grader and applied by the ledger alike: an attempt whose records show a hook or an MCP server that is not the hub's, or whose Claude transcript cannot be read, is unavailable. A turn-free attempt is valid only with its context paths working: both verified before its tasks, and none lost, no cohort lifted and none formed open while the agents worked; teardown comes after that and does not count. Whether the agents' plans overlapped, so that a cohort formed at all, is their doing after assignment and is not a condition: every turn-free attempt without a capability failure counts for the arm, and the ledger reports the treatment received and a median over treated attempts beside it. A capability failure while the agents work is the one exclusion after assignment, because #110 forbids reporting it as a turn-free run (AC3); such attempts are listed with their reasons, never dropped silently. Each attempt records its transcript's length and hash when it ends; validity and every Claude measure are read from that prefix (Claude Code may append rows after it exits: what it writes after the prefix was taken is never counted, the ledger reports its size as `late_append_bytes`, and a turn still open in the prefix leaves that agent's settlement unknown), and a prefix that changed counts as unreadable.
|
|
63
|
+
|
|
64
|
+
`scripts/benchmarks/manifest-v2-ablation-106.json` is the #106 ablation: the advisory arm against the same arm with `experiments.stale_notices: "deliver"`, which turns stale-notice dropping off and changes nothing else, on the same release and conditions. Its attempts are counted apart from the four-arm plan.
|
|
65
|
+
|
|
66
|
+
Pass `--repeat <n>` for the n-th repeat of a case (0 for the first): the arm order is row (case index + repeat) of a Williams design (0, 1, n-1, 2, n-2, ... shifted by the row), so over n consecutive rows every arm runs right before every other one once and repeats of one pair change which arm runs last. The manifest's `plan` names the release pilot (case 0, three repeats, 12 attempts, an active-time ceiling of one hour) and the study (ten cases, two repeats, 80 attempts, 6 hours 40 minutes at 300 s each, setup, grading and teardown excluded); `runner.py` refuses a plan whose attempts or ceiling do not follow from its arms, cases and repeats.
|
|
67
|
+
|
|
68
|
+
## Coordination ledger
|
|
69
|
+
|
|
70
|
+
```sh
|
|
71
|
+
python3 scripts/benchmarks/ledger.py --run /private/tmp/ahub-0124-r1 [--run /private/tmp/ahub-0124-r2 ...]
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
It reads each run record, the Claude transcript it names and the fixture's git history, and writes `ledger.json` into the first run directory with a `units` table that names the unit and coverage of every measure. Give `--run` once per directory to pool the repeats of one plan; an attempt a directory's `cohort.json` planned that wrote no record is listed as missing. Per attempt: completion (the runner's end reason and its detail, such as `wall-timeout`, and whether every task has a done); setup and active time; first candidate, completion intents, integration and check times, and each agent's settlement read from its own record (the end of its last native turn after the last done, turns started by late messages included; the last done itself when it did not work after it), and how long the agents could still write after the active time ended (`stopped_s`); usage per agent in task (to its last done on the board) and over the whole attempt: Codex turns, assistant messages, token-usage updates whose running total grew past the total before the window, and the token growth; Claude assistant messages (unique message ids), turns and the tokens their usage records; Claude's main-loop requests by request id (side requests and retries are not in its transcript), and Codex's provider requests unknown (app-server 0.159 does not send its response ids); Codex turns after its done and what started them; late replies, steered or at the next turn; the hub's fact offers, acknowledgements, bytes offered and acknowledged, build times, hook start-up times and steer round trips by path, in the task window; capability readbacks; validity by the grader's gates, and for turn-free the treatment received; held-back messages (`quiet` events) apart from [FYI] messages (which include the final [FYI] the instructions ask for); stale notices; shadow split predictions with their traces; the hooks each agent's records show, by a label that keeps paths and arguments out, with Claude's hook durations and the hub's own timing of every facts hook call; and contributions. Summaries give medians over valid completed attempts and over the (case, repeat) pairs every arm completed validly, totals over the attempts whose tasks were handed out and to which the measure applies (a Codex measure in a Claude-only arm is not counted as unknown), with the number of attempts each was unknown for, and the reasons for the rest. A repeat given twice is refused, and `--plan pilot` (or `study`) lists every planned attempt that wrote no record, whole repeats included. `ledger.json` holds code fragments from the agents' writes and local paths: keep it with the private run data and never commit it; the summary is what a verification record quotes. In-task windows end at different points by arm (a turn-free integration step comes before the done, an advisory completed-change notice turn after it), so arms are compared on whole-attempt usage.
|
|
75
|
+
|
|
76
|
+
Contributions are a heuristic for possible loss, never a certificate: per agent and file, the identifiers and changed fragments its applied writes introduced (only a Claude tool call with a successful result, or a completed Codex patch, counts; a Write replaces the agent's earlier contribution and is not credited with what other agents wrote; a delete removes it; when a file is moved, every agent's contributions to it are checked at its new path) that the final tree lacks, a fragment counting as present anywhere in the file. A same-name overwrite shows as a lost fragment. Shell commands run during the task are not attributed and are counted under `coverage`, with a missing transcript or an unreadable file. Correctness comes from the official grader, for every arm.
|
|
77
|
+
|
|
78
|
+
## Preregistration (v2)
|
|
79
|
+
|
|
80
|
+
Fixed before outcomes are collected (issue #110):
|
|
81
|
+
|
|
82
|
+
- Order of ablations: the stale-notice change first (#106: `manifest-v2-ablation-106.json`, the advisory arm with and without stale-notice dropping on the same release; comparing with 0.12.3's runs would confound it with that release's Codex hooks), then cohort silence with acknowledged facts and the revision-fenced integration (#107 and #108, the turn-free arm). The split rule (#109) is recorded as shadow predictions only; task allocation stays fixed by case index in both joint arms.
|
|
83
|
+
- Primary outcome: the final submitted artifact of each attempt, graded by the official tests with the same controls for every arm. Completed and timed-out attempts are graded (a timeout's final tree is its submission, collected once the agents were stopped, `stopped_s` after the limit); interrupted, infrastructure-failed and invalid attempts are reported unavailable with their reasons. Time to both done is reported only with completion, failures, timeouts and unavailable attempts beside it, and timing comparisons use attempts that completed in every compared arm.
|
|
84
|
+
- Resource outcomes per provider over the whole attempt, in the units the ledger names: Codex turns, usage growth and tokens; Claude assistant messages, main-loop requests and tokens (amended on 2026-10-02, before the study: side requests are not in Claude's transcript); hook and fact bytes and latencies.
|
|
85
|
+
- The turn-free arm is analysed over every attempt without a capability failure (the only exclusion after assignment, required by AC3, and listed), with the treated attempts' median as a secondary result. A 35% time and 30% usage reduction are hypotheses, not targets met; counters of different providers are not added together.
|
|
86
|
+
- Every preregistered attempt is retained, including failed and infrastructure-unavailable ones; no repeat is selected after inspecting results.
|
|
87
|
+
- The release pilot (case 0, three repeats of four arms) is feasibility evidence only. Changing the default from advisory needs paired quality and resource criteria fixed in advance, held-out repository and task instances, and comparison with both solo controls and the current advisory arm; any fact loss, missed integration, stale-generation injection or held-back workflow event is a safety result that blocks it.
|
|
55
88
|
|
|
56
89
|
## Grade and report
|
|
57
90
|
|
package/docs/events.md
CHANGED
|
@@ -14,6 +14,15 @@ marked `private: true`, and PII tasks `pii: true`.
|
|
|
14
14
|
|---|---|
|
|
15
15
|
| `envelope` | `id`, `from`, `to` (absent for broadcast), `priority`, `hop`, `kind`, `task` (task id from refs), `bytes` (UTF-8 size of the body; absent on a private envelope, whose size would say something about the PII text), `private`, `dropped` (`hop` or `fyi` when not delivered) |
|
|
16
16
|
| `overflow`, `undeliverable` | `id`, `from`, `peer` |
|
|
17
|
+
| `stale` | `id`, `from`, `peer`, `task`: a notice about the recipient's open task, dropped unsent because, right before it would have been handed over, that task was closed, had another owner or was gone (issue #106) |
|
|
18
|
+
| `quiet` | `id`, `from`, `peers`: an agent message held back from these members of a silent turn-free cohort; its other recipients got it (issue #107) |
|
|
19
|
+
| `fact` | `peer`, `id` (the offer), `files` (files whose diff it carried), `plans`, `unknown` (files whose change it showed with attribution unknown), `named` (files under a named directory it named without a diff, those counted as "N more" or held back by the PII filter included), `bytes` (the injected text), `via` (`hook` for Claude, `steer` for Codex, `done` with an integration request), `ms` (the hub's time to build it), `hookMs` (the hook process's own start-up and connect time), `accepted` (whether app-server took the steer), `unanswered` (app-server did not answer it within 10 s: it may have gone in), `rttMs` (an accepted steer: from sending it to app-server's answer), `probe` (a context check), `coverage` (it named files earlier changes are not covered for): one fact offer (issue #108) |
|
|
20
|
+
| `fact_ack` | `peer`, `id`, `via` (`hook` and `steer`: a readback found the offer in the native session; `done`: the next `hub_task_done`), `ms` (from the offer): an acknowledged offer, the only thing that moves a peer's view |
|
|
21
|
+
| `capability` | `peer`, `state` (`verified` or `lost`), `via`: a peer's context path for facts |
|
|
22
|
+
| `native_turn_end` | `peer`: Claude's Stop hook, the end of its turn (Codex's is its `turn_end`); quiescence evidence for an integration |
|
|
23
|
+
| `hook_stats` | `peer`, `n` (facts hook calls in the turn, its Stop included), `startupMs` and `maxStartupMs` (the hook processes' start-up and connect time, summed and the largest), `hubMs` (the hub's own time for them): at Claude's Stop (issue #108) |
|
|
24
|
+
| `cohort` | `id`, `event` (`formed`, `joined`, `lifted`), `silent`, `tasks`, `owners`: owners of overlapping tasks formed a cohort, it changed membership, or it stopped being silent (issue #107). Recorded in every regime; only a turn-free project's cohorts can be silent |
|
|
25
|
+
| `split` | `task`, `where` (`routing`: routing chose the first owner of a task overlapping another owner's task not started yet, not an escalation, relay or reassignment, the record calibration reads; `cohort`: an overlap formed or changed a cohort), `verdict` (`split`, `single`, `unknown`), `single` (the peer that would finish both units alone soonest), `splitS`, `singleS`, `reason` (for `unknown`), `trace` (the inputs and steps: peer names, their profiles of versions and coordination, and numbers only): a shadow split prediction; it never changes the assignment (issue #109) |
|
|
17
26
|
| `state` | `peer`, `state` |
|
|
18
27
|
| `turn_start` | `peer`, `turn` (`<peer>#<hub run>.<n>`, unique across restarts). A turn follows the adapter: pausing a busy peer does not end it |
|
|
19
28
|
| `turn_end` | `peer`, `turn`, `ms`, `tokens` (when the adapter reported any during the turn), `files` and `snapshotMs` (when snapshots are on: how many files the turn changed, and the time both snapshots took) |
|
package/docs/operations.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Operations guide
|
|
2
2
|
|
|
3
|
-
This guide describes ahub 0.12.
|
|
3
|
+
This guide describes ahub 0.12.5 and control protocol 13. Live verification
|
|
4
4
|
results and remaining prerequisites are recorded separately in [the smoke ledger](smoke.md).
|
|
5
5
|
|
|
6
6
|
## Install and start
|
|
@@ -344,6 +344,174 @@ are quoted and marked as other agents' text. The hook runs `ahub`, so it has to
|
|
|
344
344
|
be on the PATH Claude Code's hooks see; otherwise every edit shows a hook error
|
|
345
345
|
(it never blocks). `ahub check-path <file>` prints the same list in a terminal.
|
|
346
346
|
|
|
347
|
+
## Turn-free coordination
|
|
348
|
+
|
|
349
|
+
A notice about an open task of its recipient (the completed-change notice and the
|
|
350
|
+
edit-conflict messages above) is checked again for each recipient right before it
|
|
351
|
+
is handed over, after any digest condensation: if that recipient's task has
|
|
352
|
+
closed, changed owner or is gone, its copy is dropped instead of starting a turn,
|
|
353
|
+
the journal records it as `discarded`, `hub.log` has a `STALE` line and
|
|
354
|
+
`events.jsonl` a `stale` event. Other recipients keep their copies. Approvals,
|
|
355
|
+
check results, review requests, assignments and budget, permission and recovery
|
|
356
|
+
messages are never dropped this way, and a dropped notice never resolves another
|
|
357
|
+
delivery. The condition lives in memory: after a restart, or for the oldest of
|
|
358
|
+
more than 1024 notices, the notice is delivered as before.
|
|
359
|
+
|
|
360
|
+
Set `"coordination": "turn-free"` in `.agenthub/config.json` to let owners of
|
|
361
|
+
overlapping open tasks (the overlap rules above) work without messaging each
|
|
362
|
+
other. The default, `"advisory"`, keeps the behaviour described so far, and it
|
|
363
|
+
stays the default until an evaluation says otherwise (`docs/cooperbench.md`).
|
|
364
|
+
|
|
365
|
+
- Cohorts. Owners of overlapping tasks form a cohort when the overlap is found.
|
|
366
|
+
It is silent only if, at that moment, turn-free is on, no PII task is open and
|
|
367
|
+
every owner's context path is verified (below). It never turns silent later;
|
|
368
|
+
it stops being silent, for good, when an owner without a verified path joins,
|
|
369
|
+
a path is lost or a PII task opens, and every member still at work is told at
|
|
370
|
+
once that it may message again, with the completed-change notices the silence
|
|
371
|
+
held.
|
|
372
|
+
- Verified context path. A hub-launched Claude session (`ahub claude`) gets the
|
|
373
|
+
hub's hook before and after every tool call and at the end of each turn, in
|
|
374
|
+
its `--settings` next to the status line tee (a `--settings` of your own turns
|
|
375
|
+
them off). Codex gets context by steer into its running turn. Until a fact or a
|
|
376
|
+
one-line probe has been read back, the peer counts as unverified: for Claude
|
|
377
|
+
the row Claude Code writes in its transcript for the hook's additional
|
|
378
|
+
context, matched by tool use id and the offer's id; for Codex the steered
|
|
379
|
+
input coming back as a user message item of the turn. A new native session,
|
|
380
|
+
the peer going offline, or three offers past a minute without a readback
|
|
381
|
+
(checked at its boundaries and every 30 seconds; facts sent with an
|
|
382
|
+
integration request wait for the next done and do not count, a refused steer
|
|
383
|
+
is no offer, and an unanswered one may still be read back), makes it
|
|
384
|
+
unverified again, and a Claude session whose
|
|
385
|
+
transcript the hub cannot find gets no facts at all and loses a verified
|
|
386
|
+
path. A session whose hooks stop altogether leaves no offer unread, so it
|
|
387
|
+
stays verified; a peer without a verified path that left three
|
|
388
|
+
offers unread gets none until a readback arrives. Kimi, Pi and the local
|
|
389
|
+
worker have no path yet, so a cohort with one of them is never silent.
|
|
390
|
+
- Silence. While a cohort is silent, an agent message from a member to another
|
|
391
|
+
member is held back for that recipient only: other recipients and the console
|
|
392
|
+
get it unchanged, and `events.jsonl` records a `quiet` event. `hub_send`
|
|
393
|
+
answers `not delivered to <peer>: ...` (or `sent to: ...; not delivered to
|
|
394
|
+
...` when some recipients got it); a native turn answer's sender hears it on
|
|
395
|
+
its next delivery. Workflow messages from the hub are never held back. A
|
|
396
|
+
member's messages stay held until its native turn has ended after its task
|
|
397
|
+
closed (Codex's turn completes, or Claude's Stop hook runs; a member already
|
|
398
|
+
between turns when its task closes has stopped; a paused peer may still be in
|
|
399
|
+
its turn, and Claude's channel going offline says nothing about its session),
|
|
400
|
+
so a late answer is still the cohort's. That settlement is recorded when it happens and
|
|
401
|
+
never undone: the member's next turn is new work, and once every member has
|
|
402
|
+
settled the cohort is over (a task reopened after that is outside it). Only a tool call starting counts as activity after
|
|
403
|
+
a turn end. A message held back from all its recipients does not count
|
|
404
|
+
against the sender's limits.
|
|
405
|
+
- Facts. At each tool call (Claude) or completed tool item (Codex) of a cohort
|
|
406
|
+
member with an open task, the hub offers what changed since it last
|
|
407
|
+
acknowledged them in the files every member's task names and in the files it
|
|
408
|
+
touched, with the other members' new plans; the last member still at work
|
|
409
|
+
keeps the others' files after they finish. A directory a task names stands for
|
|
410
|
+
git's changed (staged or not) files against HEAD, new and deleted files under
|
|
411
|
+
it (200 at most; the fact says when more were cut, until it is read back, and a
|
|
412
|
+
new session hears it again). A file the peer has seen there stays covered after git
|
|
413
|
+
stops listing it (put back to HEAD's bytes, or the directory moved away), so
|
|
414
|
+
the way back is shown (200 at most, the newest versions first; the rest are
|
|
415
|
+
named as no longer followed, until the peer reads that back; a new session
|
|
416
|
+
hears only the drops of its own). A file there that the peer has neither seen nor touched appears once
|
|
417
|
+
someone changed it so that its bytes differ from HEAD's, or created it: it is
|
|
418
|
+
named without a diff (what happened before is never shown) until the peer reads
|
|
419
|
+
the fact back, and until then it counts as a change the peer has not been
|
|
420
|
+
shown. A rewrite with HEAD's bytes, which git lists until it refreshes its
|
|
421
|
+
index, is no change (a file too large to read is compared by git's own hash).
|
|
422
|
+
`.git` directories
|
|
423
|
+
at any depth and what the denylist keeps from every agent
|
|
424
|
+
(`src/local/deny.ts` and `local.deny`) are never read or shown. A file a peer touched before it had a view of it (a
|
|
425
|
+
partial read, say) is compared with what it was then, so a change landing in
|
|
426
|
+
between is shown. A change is credited to an agent
|
|
427
|
+
only with effect evidence: a Claude Edit, MultiEdit or Write whose result is
|
|
428
|
+
exactly its input applied to the file as observed before it, or a Codex patch
|
|
429
|
+
whose diff is exactly what changed. Shell commands, concurrent writers and
|
|
430
|
+
unreported changes are shown with their attribution unknown, never credited by
|
|
431
|
+
elimination; an agent's own verified writes are not shown back to it. A Codex
|
|
432
|
+
read action never counts as having seen a file (it may be partial); a Claude
|
|
433
|
+
Read does when it returns the whole file: no offset or limit, at most 2000
|
|
434
|
+
lines and no line over 2000 characters. A diff that matches a PII pattern is
|
|
435
|
+
not shown (the file is named, to be read), and a changed file whose name
|
|
436
|
+
matches one is counted, not named. Only an acknowledgement (a readback, or the next
|
|
437
|
+
`hub_task_done` for facts sent with an integration request) moves the peer's
|
|
438
|
+
view, so a fact that does not arrive is
|
|
439
|
+
offered again at a later boundary; no turn is ever started for one. An
|
|
440
|
+
acknowledgement says the context reached the native session, not that the
|
|
441
|
+
model read it. 60 changed lines are shown at most, the cut files named; a
|
|
442
|
+
history longer than the hub keeps is shown with its attribution unknown, and a
|
|
443
|
+
file that falls out of the 64 a peer touched is named until the peer reads
|
|
444
|
+
that back. Only regular files
|
|
445
|
+
of 256 KB or less inside the project are read, re-resolved at every read and
|
|
446
|
+
opened without following links; larger ones are named without a diff. Facts never go through the bus or the delivery journal;
|
|
447
|
+
`events.jsonl` records `fact`, `fact_ack` and `capability` events with bytes
|
|
448
|
+
and latencies.
|
|
449
|
+
- Integration. A member's `hub_task_done` is a completion intent. The member
|
|
450
|
+
whose intent completes the set is asked, as its done result, to check its work
|
|
451
|
+
against the others' (their files, signatures and summaries, plus its own facts)
|
|
452
|
+
and to call `hub_task_done` again; nothing is recorded as done yet. The next
|
|
453
|
+
call counts only for the same target: the same owner, cohort revision and
|
|
454
|
+
files (the named ones, and those each member wrote with an edit tool the hub
|
|
455
|
+
saw, Claude's Edit, MultiEdit and Write or a Codex patch, between being handed
|
|
456
|
+
its task and settling, so a symbol-only overlap counts and
|
|
457
|
+
a member settling keeps its files in; reading a file never moves it, a settled
|
|
458
|
+
member's later work does not count, a shell command's writes outside the named
|
|
459
|
+
paths are not seen, and a file the hub reads whole whose bytes equal HEAD's
|
|
460
|
+
is no change, whatever git's stat data says), with every other owner's native turn
|
|
461
|
+
ended after its done (or that owner idle when it finished); the integrating
|
|
462
|
+
owner's own other tasks in the cohort never count as still running. A done of a member by the console counts as its
|
|
463
|
+
intent too. Edits in between, a new member, an owner change, a failed check or
|
|
464
|
+
a reopened review ask again; a done within two seconds of a request is taken as a retry and gets the
|
|
465
|
+
same request again; after three requests the done is recorded with
|
|
466
|
+
`integration unresolved`, never as integrated, and that revision asks nothing
|
|
467
|
+
more (a configured check then counts as usual). A configured check of the integrating member
|
|
468
|
+
counts only for the target it confirmed; when another member reopens its task,
|
|
469
|
+
that member integrates instead and the earlier one's check counts as usual.
|
|
470
|
+
Inside a silent cohort a member's completed-change notice is held, not sent;
|
|
471
|
+
where no integration step runs for the others (the silence was lifted, a PII
|
|
472
|
+
task opened), the held notices go with the lift notice or the done result, and
|
|
473
|
+
for a task the console finishes, to the console. Open tasks outside the cohort
|
|
474
|
+
get their notices as usual. Facts sent with an integration request are
|
|
475
|
+
offered again at the next boundary until the next done acknowledges them.
|
|
476
|
+
Cohorts live in memory: after a hub restart an open
|
|
477
|
+
request is recorded as unresolved, and when a peer first attaches, each of its
|
|
478
|
+
open tasks that overlaps other work hears that overlaps are settled by message
|
|
479
|
+
again, with the completed-change notices of overlapping tasks finished since
|
|
480
|
+
it was handed the task (a notice may come twice; none is lost).
|
|
481
|
+
- While any PII task is open the project behaves as advisory: no facts, no
|
|
482
|
+
silence and no integration step, and the cohorts that were silent stay lifted.
|
|
483
|
+
When it closes, the hub forgets what it had observed, so nothing changed
|
|
484
|
+
meanwhile is shown as a diff; each member is told which of its files to read
|
|
485
|
+
again.
|
|
486
|
+
- `ahub check-path` asks the hub whether the owner of the claimed path shares a
|
|
487
|
+
silent cohort with the caller, and only then leaves out the request to settle
|
|
488
|
+
by message. `templates/claude-hooks.json` holds only the check-path hook; the
|
|
489
|
+
facts hooks need a hub-launched session.
|
|
490
|
+
- Routing does not change. When routing chooses the first owner of a task that
|
|
491
|
+
overlaps another owner's task not started yet (`where: "routing"`, the record
|
|
492
|
+
calibration reads; an escalation, relay or reassignment is not one), and when an overlap forms or changes a cohort, routed or
|
|
493
|
+
named (`where: "cohort"`), the hub records a shadow split prediction (`split`
|
|
494
|
+
event; `ahub route explain <id>` shows its trace as it would be now): whether
|
|
495
|
+
splitting two equal units between the two peers (`o_s + u_s < o_f + 2u_f`)
|
|
496
|
+
would finish sooner than the faster one alone, from the recorded task stages of
|
|
497
|
+
each peer under the profile it has now. Every hand-over is tagged with the new
|
|
498
|
+
owner's profile: the hub's version, the agent's (Codex's from app-server,
|
|
499
|
+
Claude Code's from its transcript; other agents report none yet) and the
|
|
500
|
+
coordination mode, which decides the hub's own hooks; records accumulate across
|
|
501
|
+
hub runs, one per hand-over (a decline or an escalation away counts against
|
|
502
|
+
the peer that failed, never the next owner), and a peer whose version is
|
|
503
|
+
unknown has no profile. The user's and
|
|
504
|
+
plugins' hooks are not part of it. It is unknown unless the units are equal
|
|
505
|
+
and known, both peers are available (the other owner idle, or busy taking the
|
|
506
|
+
overlapping task while it is not started; the task's own peer idle or busy
|
|
507
|
+
taking it) with no other open work (an overlapping task its owner
|
|
508
|
+
has started counts), and each has five measured tasks with no more than 30%
|
|
509
|
+
failures and comparable work times. The work stage of a task ends at its first
|
|
510
|
+
`hub_task_done`, and the task itself is never one of its own observations.
|
|
511
|
+
- `"experiments": {"stale_notices": "deliver"}` in `.agenthub/config.json`
|
|
512
|
+
turns the stale-notice drop off, for a controlled comparison only (the #106
|
|
513
|
+
ablation in `docs/cooperbench.md`); the hub logs it at start.
|
|
514
|
+
|
|
347
515
|
## Approvals and pauses
|
|
348
516
|
|
|
349
517
|
Inspect permission requests in the terminal:
|
|
@@ -474,21 +642,21 @@ Rows without a live process are stale registrations; forget them with
|
|
|
474
642
|
|
|
475
643
|
Upgrade running projects with the target release's own coordinator. It accepts
|
|
476
644
|
a running source on control protocol 9 (0.6.x), 10 (0.7.0 through 0.12.0),
|
|
477
|
-
|
|
645
|
+
11 (0.12.1 and 0.12.2), 12 (0.12.3) or 13 (0.12.4) and only
|
|
478
646
|
a target on its own protocol, so the target's coordinator fits every supported
|
|
479
647
|
source and carries every recovery fix released up to it. Protocol 8 and older
|
|
480
648
|
(0.5.x and earlier) are refused as `manual-bootstrap-required`. Run from the
|
|
481
649
|
project directory, without replacing the global CLI first:
|
|
482
650
|
|
|
483
651
|
```bash
|
|
484
|
-
bunx --package @staix/agent-hub@0.12.
|
|
485
|
-
bunx --package @staix/agent-hub@0.12.
|
|
652
|
+
bunx --package @staix/agent-hub@0.12.5 ahub upgrade --to 0.12.5 --dry-run
|
|
653
|
+
bunx --package @staix/agent-hub@0.12.5 ahub upgrade --to 0.12.5 --yes
|
|
486
654
|
```
|
|
487
655
|
|
|
488
656
|
| Running now | Coordinator to use |
|
|
489
657
|
| --- | --- |
|
|
490
658
|
| 0.6.x (protocol 9) | the target's, through `bunx` as above |
|
|
491
|
-
| 0.7.0 through 0.12.0 (protocol 10) | the target's, through `bunx` as above |
|
|
659
|
+
| 0.7.0 through 0.12.0 (protocol 10), 0.12.1 and 0.12.2 (protocol 11), 0.12.3 (protocol 12), 0.12.4 (protocol 13) | the target's, through `bunx` as above |
|
|
492
660
|
| any supported source, with the installed CLI already at the target | `ahub upgrade` below, which is the same coordinator |
|
|
493
661
|
| 0.5.x or earlier (protocol 8 and older) | not supported: bootstrap by hand with the matching CLI |
|
|
494
662
|
|
|
@@ -1187,4 +1187,34 @@ of a native turn boundary. Failed notifications, disconnects, replaced sessions
|
|
|
1187
1187
|
and interrupted daemon instances retain `needs_review`, with no automatic replay.
|
|
1188
1188
|
A person inspects and resolves uncertain work through the existing revision-fenced
|
|
1189
1189
|
`ahub queue` commands. Older source protocols 9, 10 and 11 remain authenticated
|
|
1190
|
-
upgrade sources; ordinary clients must use protocol 12.
|
|
1190
|
+
upgrade sources; ordinary clients must use protocol 12 (13 from 0.12.4, with 12 a source).
|
|
1191
|
+
|
|
1192
|
+
## Amendment: stale overlap notices (issue #106)
|
|
1193
|
+
|
|
1194
|
+
- The completed-change notice (#31) and the edit-conflict notices (#32, #91) are about an open task of their recipient. `Tasks.whileOpen` publishes each to one recipient and remembers the condition (the newest 1024). The bus asks `Tasks.relevant(peer, env)` per recipient when it builds a delivery and again right before it hands the delivery over, after condensation; it is false only for a recorded notice to that peer whose task is gone, has another owner, or left the states it was about: `proposed`, `in_progress`, `changes_requested`, and for conflict notices also `in_review` (#91 reports conflicts with work under review). A dropped copy is a journal row `discarded` with `stale: task #N is no longer open for <peer>`, a `STALE` log line and a `stale` event; other recipients and other deliveries are untouched.
|
|
1195
|
+
- An envelope without a record (another kind, a restart, an evicted record) is delivered as before: its purpose is never guessed from its kind. Assignments, approvals, check results, review requests and budget, permission and recovery messages are never conditional.
|
|
1196
|
+
- The ablation is the advisory arm with and without stale-notice dropping on the same release (`manifest-v2-ablation-106.json`, #110); comparing with 0.12.3's advisory runs would confound it with that release's Codex hooks.
|
|
1197
|
+
|
|
1198
|
+
## Amendment: turn-free cohorts (issue #107)
|
|
1199
|
+
|
|
1200
|
+
- `coordination` in the project config: `"advisory"` (default) or `"turn-free"`; anything else is advisory with a log line. While a PII task is open the project is advisory.
|
|
1201
|
+
- `src/hub/cohorts.ts`: owners of overlapping open tasks (#31) form a cohort when the overlap is found (assignment, an accept with a plan). Members are tasks with their owner and owner generation (ownership events); every change of membership or owner, a withdrawn intent and a lift bump the cohort's revision. A cohort is silent if, when formed, every owner's context path is verified (#108); it never becomes silent later, and it is lifted for good (members still at work told at once, with the held notices) when an unverified owner joins, a path is lost or a PII task opens. Cohorts live in memory: after a restart, each peer's first attach tells its open overlapping tasks that overlaps are settled by message again and replays the completed-change notices of overlapping tasks finished since they were handed over.
|
|
1202
|
+
- Silence is a per-recipient bus policy (`BusOptions.silence`), applied once the audience is final (implicit replies and `digest` resolved): an agent's chat from a member that has not settled to another member is not queued for that member; the other recipients get the envelope unchanged, so correlation, priority caps, dedupe and hops are untouched. A `quiet` bus event goes to the log and `events.jsonl`; `hub_send` answers with the held-back peers (an error when nobody got it); a native turn answer's sender hears it as a note. A member settles once its task left the open states and its native turn has ended after that (Codex's adapter leaving its turn, Claude's Stop hook; a member already between turns when its task closes settles at once; a paused peer counts by its adapter's state, and Claude's channel going offline says nothing about its session). Settlement is recorded as it happens (`Cohorts.turnEnded`, `Cohorts.closed`) and never undone: a settled member's next turn is new work, and a cohort whose members all settled is over. A message held back from all its recipients is not admitted against the sender's limits.
|
|
1203
|
+
- Texts: in a silent cohort, overlap results and notices, task envelopes and the conflict notices name the plans and say not to message; `hub_task_accept` returns the overlapping plans with every accept; `ahub check-path` asks the hub (`silenced` control request). A member's completed-change notice to another member is held (`Cohort.held`), never dropped: it goes with the lift notice when silence is lifted, with the done result of a member for whom no integration step runs (turn-free off, a PII task open), or to the console when the console finishes the task (its done still counts as that member's intent). Notices to open tasks outside the cohort go as usual. Otherwise the advisory texts stay.
|
|
1204
|
+
- Integration: a member's `hub_task_done` records an intent. The member whose intent completes the set is selected in one synchronous step and gets an integration request as its done result (the others' files, signatures and summaries, plus its facts); the request is recorded as `integration requested`, not done. Its next done is accepted (`integrated`, then the usual done) only for the same owner generation, cohort revision and file hash (named files, for a directory git's changed, new and deleted files under it against HEAD, and the files each member wrote with an edit tool the hub saw (Claude's Edit, MultiEdit and Write; Codex's patches) between being handed its task and settling; reads never count, nor shell writes outside the named paths, and a file the hub reads whole whose bytes equal HEAD's is no change there), with its facts acknowledged (the done first acknowledges the facts sent with the request) and every other owner stopped after its intent (a native turn end after it, or idle when it was recorded); the integrating owner's own other tasks never count as running. A cohort whose members have all settled is over, whoever asks first, and is never lifted or reopened after that. A done within two seconds of a request is a retry and gets the same request, with nothing acknowledged or counted. Otherwise it is asked again; past three requests the done is recorded with `integration unresolved`, and that revision asks nothing more (the member's check counts as usual). A failed check or changes requested withdraws that member's intent; a check of the integrating member that passes after its target moved within the same revision counts as finished late; once the revision moved (another member reopened), the former integrating member is an earlier finisher and its check counts. After a restart an open request is recorded as unresolved.
|
|
1205
|
+
|
|
1206
|
+
## Amendment: turn-free facts at tool boundaries (issue #108)
|
|
1207
|
+
|
|
1208
|
+
- `src/hub/facts.ts` keeps three things apart per peer: what the hub observed in the tree (latest version and transitions per file), what it offered (bounded offers with their file versions and plans), and what the peer acknowledged (its view). Only an acknowledgement moves a view; a later boundary offers everything since the view again. Files are the paths every task of each live cohort in which the peer has an open task names (a directory expands to git's changed files against HEAD, staged or not, and its new and deleted files, 200 at most, with a note when more were cut, told until it is read back and again in a new session), contained in the project as real paths and never `.git` or anything the denylist (`src/local/deny.ts`, `local.deny`) keeps from agents, plus the last 64 it touched in its current session; a file that falls out of those is named in every offer until one naming it is read back (#112). A file the peer touched before it had a view of it is compared with its version at that touch, so a change in between is shown, not absorbed into a first look; a file it saw under a named directory stays covered after git stops listing it (put back to HEAD's bytes, or the directory moved away; 200 at most, the rest named as no longer followed until that is read back; a new session hears only the drops of its own). A directory file it has neither seen nor touched is named without a diff when it appears with bytes that differ from HEAD's (a rewrite with HEAD's bytes, listed until git refreshes its index, is no change, for the fact and for whether the peer is current; a file too large to read is compared by git's blob id of its bytes, hashed in-process, while all such files in one comparison total 64 MiB or less, and is not compared otherwise (so the answer never depends on what an earlier comparison hashed); the bytes are the raw working-tree bytes, without git's filters, so with an end-of-line conversion or LFS such a file reads as changed and is named; #112) (diffing it against HEAD would show what changed while tracking was off, a PII window, and credit unseen changes by elimination), and counts as not yet shown until that fact is read back. Only regular files of 256 KB or less are read, re-resolved at every read and opened without following a final link or blocking. A history longer than the 64 kept transitions makes the change's attribution unknown.
|
|
1209
|
+
- Attribution: a transition is credited only with effect evidence: a Claude Edit, MultiEdit or Write whose result equals its input applied to the file as observed at PreToolUse with no observation in between, or a Codex `fileChange` whose changed lines equal the observed diff. Everything else is shown with its attribution unknown; an agent's own verified writes advance its view and are not shown back. A Claude Read advances its view only when it returned the whole file (no offset or limit, at most 2000 lines, none over 2000 characters); a Codex read action never does (it may be partial). A diff that matches a PII pattern is replaced by a note naming the file, and a changed file whose name matches one is counted without its name.
|
|
1210
|
+
- Acknowledgement and capability: Claude's hook runs before and after every tool and at Stop (`ahub claude` injects it in a turn-free project); the pre answer's text becomes `additionalContext`, and the row Claude Code writes in its transcript for it (matched by tool use id and offer id) is the readback. Codex's offer goes in with `steerText` (bound to the running turn), and the steered input coming back as a `userMessage` item is the readback. A readback marks the peer's path verified; a new session or thread, the peer going offline, or three offers past a minute without one (checked at boundaries and every 30 seconds; facts sent with an integration request wait for the next done and never count, a refused steer is dropped, not unread, and an unanswered one stays pending, since it may have gone in) unverifies it. Until verified, a peer gets a one-line probe at most three times per session, and no offers at all once three are unread; a Claude session whose transcript the hub cannot find gets none at all and loses a verified path.
|
|
1211
|
+
- The control contract is protocol 13: `facts` carries the phase, tool, input, tool use id, session id and transcript path; `silenced` serves check-path; `send` results name held-back recipients. Protocol 12 stays an authenticated recovery source.
|
|
1212
|
+
- While a PII task is open nothing is observed or offered; every board change re-checks this, so tracking switches at once. When tracking resumes everything observed is dropped and each peer is told which files earlier changes are not covered for. Turn ends and tool starts are recorded in every regime.
|
|
1213
|
+
- `events.jsonl`: `fact` (bytes, build time, the hook's own time, the steer round trip), `fact_ack` (latency), `capability`, `native_turn_end`, `hook_stats` (every facts hook call of a Claude turn, at its Stop), `cohort`.
|
|
1214
|
+
|
|
1215
|
+
## Amendment: shadow split prediction (issue #109)
|
|
1216
|
+
|
|
1217
|
+
- `predictSplit(input)` is a pure function next to `assign()`, which no longer takes a split option: assignment never changes. For the pair formed by the peer routing picks and the owner of an overlapping open task, it compares a split (the later of each peer's orientation plus one unit) with the best peer alone (orientation plus two units). It is unknown unless the units are equal and known (one task of the same class is one unit), both peers are available (the other owner idle, or busy taking the overlapping task while it is not started, as an owner goes busy when its task is delivered; the task's own peer idle or busy taking it) with no other open work (an overlapping task its owner has started counts), and each has five measured tasks with its current profile and at most 30% failures and work times whose IQR does not exceed their median; a difference under a tenth of the single time is inconclusive.
|
|
1218
|
+
- `Tasks.splitObservations` reads one observation per hand-over to the peer with the profile it has now, across hub runs of the project (what happened from that hand-over up to and with the next one is that peer's: a decline or escalation away is its failure, never the next owner's), never the routed task itself, leaves out claims, types failures (failed check, changes requested, escalation, release, decline, unresolved integration), ends the work stage at the first done call (checks and integration are not work) and leaves the stages unknown for an accept recorded by the done itself. The `split` event carries the trace.
|
|
1219
|
+
- An overlap that forms or changes a cohort records a `split` event for the task, routed or named (a benchmark names every owner); `ahub route explain <id>` appends the trace as it would be now. Rerouting needs held-out evidence through #110 first.
|
|
1220
|
+
- 2026-10-03 (T3 calibration data, issue #109 amendment): every hand-over records the new owner's split profile on its history entry: the hub's version, the agent's version (Codex's from app-server's `initialize` answer, Claude Code's from its transcript rows) and the hook profile, which is the hub's own (the coordination mode: a turn-free project runs the facts hooks in Claude). Observations count for a prediction only with the peer's current profile; while a version is unknown there is no profile, and the prediction is unknown. The prediction calibration reads is the one recorded when routing chose the first owner (no single named candidate, no claim; not an escalation, budget relay or reassignment of work already begun) of a task that overlaps another owner's task not started yet (`where: "routing"`), and `route explain` prefers the same unstarted pair; cohort-time predictions stay in the log with `where: "cohort"` and are not calibration data. The user's and plugins' hooks are not seen by the hub (ponytail: read them from the native records as the benchmark ledger does), and only Claude and Codex report a version so far.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Agent-hub 0.12.4 verification
|
|
2
|
+
|
|
3
|
+
Implementation and candidate verification for issues #106 through #110, performed on 2026-10-02. The release pilot ran on runtime and runner source `ae8f8f5`; one more four-arm repeat ran on `3555ed5`, after the later review fixes. The released source differs from `3555ed5` in the facts for files under a named directory (no live task named a directory), a behaviour-preserving refactor in `tasks.ts` and `native.ts` (the latter changes the runner hash, so that repeat cannot be re-graded by the released runner), the fact event type, the ledger, tests and docs. Production application is verified separately after publication.
|
|
4
|
+
|
|
5
|
+
## Runtime checks
|
|
6
|
+
|
|
7
|
+
- Recovery: a real published 0.12.3 package (protocol 12) was recovered into the candidate (protocol 13) with `scripts/smoke-recovery-09-10.ts` at `ae8f8f5`, its queued envelope ids preserved (operation `975034f1-b2c0-4aba-9ba9-611ef1ce5daa`). The control protocol did not change after it.
|
|
8
|
+
- Context paths: in every turn-free attempt and in the setup-only calibration, both native readbacks verified before the first task. Claude Code 2.1.287 wrote the hook's additional context as a transcript row matched by tool use id and offer id; Codex 0.159.3 returned the steered fact as a user message item of the running turn.
|
|
9
|
+
- Facts reached the owners and were read back: 17 offers in the three pilot attempts, 37,936 bytes, all acknowledged; 5 more (all acknowledged) in the repeat on `3555ed5`. No attempt lost a context path while the agents worked.
|
|
10
|
+
- Isolation: no attempt ran a Codex hook or a Claude hook other than the hub's facts hook. Every Codex started only the hub's MCP server: its wrapper turns off the user's plugins, apps, sub-agents and turn-end notifier and disables each MCP server of the user's configuration by name. In the repeat on `3555ed5` Claude's transcript shows the fixture instructions it received through `--append-system-prompt-file`.
|
|
11
|
+
|
|
12
|
+
## Native CooperBench release pilot
|
|
13
|
+
|
|
14
|
+
Feasibility only (#110, T9): one pair (`pallets_click_task:2068`, features 1 and 6), three repeats of the four v2 arms and three repeats of the #106 ablation, graded by the official tests. No adoption claim is made; advisory stays the default.
|
|
15
|
+
|
|
16
|
+
- Official source commit: `63b9d44d9f39a02fccf5bf0052db48a917a011fd`.
|
|
17
|
+
- Codex CLI 0.159.3, `gpt-6.1-sol`, medium effort; Claude Code 2.1.287, `claude-opus-5-5`, medium effort.
|
|
18
|
+
- Each feature-work arm has a 300-second wall limit; setup and denied hidden-file probes are unscored. The arm order is a Williams design, a different row per repeat.
|
|
19
|
+
- The official evaluator runs on the recorded image digest, with empty-base-fail and combined-oracle-pass controls; every control passed.
|
|
20
|
+
|
|
21
|
+
Seconds to both tasks done (active time in parentheses), and the official grade:
|
|
22
|
+
|
|
23
|
+
| Repeat | Solo Codex | Solo Claude | Advisory | Turn-free | Grading |
|
|
24
|
+
|---|---|---|---|---|---|
|
|
25
|
+
| 0 (`ae8f8f5`) | 174.6 (183.9) | 43.6 (49.6) | 176.3 (189.3) | 161.3 (172.2) | 4/4 passed both features |
|
|
26
|
+
| 1 (`ae8f8f5`) | 189.1 (198.3) | 54.3 (59.9) | 153.2 (165.9) | 168.8 (178.7) | 4/4 passed |
|
|
27
|
+
| 2 (`ae8f8f5`) | 170.6 (179.3) | 39.9 (45.1) | 173.3 (186.9) | 151.8 (161.1) | 3/4 passed; Solo Claude unavailable |
|
|
28
|
+
| 3 (`3555ed5`) | 154.9 (164.1) | 39.2 (45.1) | 153.6 (167.4) | 147.2 (156.5) | 4/4 passed |
|
|
29
|
+
|
|
30
|
+
Repeat 2's Solo Claude attempt completed, but its grade is unavailable: Claude Code wrote its last answer and a turn-duration row about 0.1 s after the attempt's transcript hash was taken, so the integrity check reported "transcript changed since the attempt". Attempts now record the transcript's length with its hash and read that prefix only (`3555ed5`; repeat 3 graded under it). The teardown starts five seconds after the hub sees both tasks done and every peer idle, which a final answer still being written can outlast; that answer is then not counted, and the agent's settlement is unknown. The pilot's prepared runner hashes pin the earlier check, so repeat 2 is not re-graded, and its prepared hashes were not rewritten.
|
|
31
|
+
|
|
32
|
+
Coordination over the three pilot repeats (ledger medians over valid attempts; whole-attempt usage per agent; counters of different providers are never added):
|
|
33
|
+
|
|
34
|
+
| Measure | Advisory | Turn-free |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| Both done, median s | 173.3 | 161.3 |
|
|
37
|
+
| Settlement (all agents stopped), median s | 181.7 | 167.0 |
|
|
38
|
+
| Codex turns / token-usage growth / tokens, median | 2 / 21 / 835,130 | 1 / 13 / 494,408 |
|
|
39
|
+
| Claude assistant messages / tokens, median | 8 / 213,333 | 8 / 211,666 |
|
|
40
|
+
| Codex `hub_send` calls, total | 10 | 0 |
|
|
41
|
+
| Codex turns after its done, total | 3 | 0 |
|
|
42
|
+
| Claude messages reaching Codex after its done, total | 6 | 0 |
|
|
43
|
+
| Stale notices dropped (#106), total | 3 | 0 |
|
|
44
|
+
| Integration requests / unresolved, total | 0 / 0 | 3 / 0 |
|
|
45
|
+
| Turn-free treatment received (silent cohort) | - | 3 of 3 |
|
|
46
|
+
|
|
47
|
+
Every turn-free attempt took one integration request and confirmed it; none was unresolved. Each advisory post-done turn was started by a late reply from Claude. Repeat 3, on `3555ed5`, matches the pattern: turn-free one integration request and no post-done turn; advisory one post-done turn after two late replies.
|
|
48
|
+
|
|
49
|
+
#106 ablation (advisory with and without stale-notice dropping, three repeats each, all graded and passed both features):
|
|
50
|
+
|
|
51
|
+
| Measure | Dropping on | Dropping off |
|
|
52
|
+
|---|---|---|
|
|
53
|
+
| Both done, median s | 153.1 | 158.6 |
|
|
54
|
+
| Stale notices dropped, total | 3 | 0 |
|
|
55
|
+
| Codex turns after its done, total | 3 | 3 |
|
|
56
|
+
| What started them | a late reply | a late reply with the stale notice |
|
|
57
|
+
| Codex tokens, median | 737,127 | 864,458 |
|
|
58
|
+
|
|
59
|
+
The drop removed the stale notice from every post-done turn, but in this pair the turn still happened because a late reply arrived with it. In the turn-free arm neither occurred in the pilot (no late reply, no post-done turn).
|
|
60
|
+
|
|
61
|
+
Contribution heuristic: three attempts (one turn-free, one advisory, one ablation) flagged one changed fragment each as absent from the final tree; all three passed the official grading. The heuristic cannot tell a rewrite from a loss, and a passing grade shows only that the features' tests pass. No identifier was flagged. Shell commands' writes are not attributed (each attempt lists them under coverage).
|
|
62
|
+
|
|
63
|
+
This is a one-pair shared-workspace sample with three repeats per arm: time differences of this size are within its noise, and no performance, resource or causal claim is made. Native session counters include setup probes only where the ledger says so; prompts, conversations, transcripts, private inputs, gold solutions and `ledger.json` (which holds code fragments and local paths) stay outside version control.
|
|
64
|
+
|
|
65
|
+
## Automated gate and review
|
|
66
|
+
|
|
67
|
+
`scripts/check.sh` on the released source: 689 tests passed, 0 failed, 3,506 expectations across 65 files; typecheck, bundle freshness, npm package contents and process ownership passed; `check: OK`. Linux and macOS CI passed on the final head. Five earlier heads' macOS runs failed, all on timing: the end-to-end test's steer window at `ae8f8f5` (its fake now leaves a wider one), a large-directory test's 5 s timeout at `3555ed5` (given a longer one), and two known flaky tests, a permission test whose 200 ms permission timeout a slow runner outlasts (`03122f9`, `c822f5c`, `12071c4`) and a sidecar test's 5 s timeout (`c822f5c`).
|
|
68
|
+
|
|
69
|
+
Independent host Claude Code OCR delegation reviews (read-only reviewers with the OCR rule groups, `REVIEW.md` and the `AGENTS.md` invariants) covered the runtime and benchmark changes in seven rounds, each on the exact head, and this document as well. Every Critical and Important finding was fixed and re-reviewed; the last code round found none. Changes between `ae8f8f5` and `3555ed5` were checked by tests and the repeat on `3555ed5`; later runtime changes by tests only, not by a second full pilot.
|
|
70
|
+
|
|
71
|
+
What the rounds changed, in brief:
|
|
72
|
+
|
|
73
|
+
- Settlement is recorded when it happens and never undone by a later turn (only reopening the task clears it); a paused or channel-offline peer is not taken as stopped; a finished cohort is never lifted at teardown.
|
|
74
|
+
- The integration target covers the named paths and what each member wrote with an edit tool the hub saw (Claude's Edit, MultiEdit and Write; Codex's patches) between its hand-over and its settlement (never what it only read, and a shell command's writes only under the named paths); an unresolved outcome closes its revision.
|
|
75
|
+
- Facts never read `.git` or denylisted paths; a diff matching a PII pattern is not shown; integration-request facts and refused steers never count as unread; a file under a named directory that the peer never saw is named without a diff, never compared with HEAD (one round's HEAD comparison could have shown, after a PII window, what was written during it, and was reverted).
|
|
76
|
+
- Benchmark validity gates are shared by the grader and the ledger: capability failures while the agents work exclude an attempt, teardown does not, and Codex and Claude run with hook and MCP isolation; a wrapper in each fixture turns off Codex's plugins, apps, sub-agents and turn-end notifier and disables the MCP servers of the user's configuration (Codex still reads the user's global `AGENTS.md` and can use the user's Codex skills, the same in every arm).
|
|
77
|
+
|
|
78
|
+
Known limits of 0.12.4 (minor findings of the last code round, accepted, to be filed as a follow-up):
|
|
79
|
+
|
|
80
|
+
- Seen files under a named directory beyond the 200 kept are named under the "no longer tracked" notice, whose stated reason is the 64-file touched limit; that notice is spent when an offer is built, not when it is read back, and a new session does not hear it again.
|
|
81
|
+
- A file under a named directory that is rewritten with identical bytes is named as changed to a member that never saw it until git refreshes its index; during an integration this can cost one more request, or, after the third request, record the outcome as unresolved.
|
|
82
|
+
- An offer that only names directory files reports zero files in its fact event.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Agent-hub 0.12.5 verification
|
|
2
|
+
|
|
3
|
+
Implementation and candidate verification for issues #113, #112 and #109 (T3 groundwork), performed on 2026-10-03. Native CLIs: Codex 0.160.0 (it replaced 0.159.3 on this machine during the work), Claude Code 2.1.288; the v2 manifests pin both. Production application is verified separately after publication.
|
|
4
|
+
|
|
5
|
+
## #113: the leftover after an interrupted arm
|
|
6
|
+
|
|
7
|
+
Cause, reproduced before any fix: a fixture hub with the real Codex, a Codex turn running `sleep 40`, torn down as the runner did (pause, close, `ahub kill`, `projects remove`). `codex` is a node launcher (`codex.js`) that forwards SIGTERM to the native app-server and waits; mid-turn the app-server does not exit, so after 1 s the hub SIGKILLed the launcher alone. The app-server went on under init (PPID 1) with its MCP servers, holding the launcher's stderr pipe, and the daemon, its state removed and its claim released, stayed alive on that pipe: `kill` answered "hub is not running". With the app-server in its own process group, stopped as one, nothing of the fixture was left and `kill` took 173 ms instead of 1,087 ms.
|
|
8
|
+
|
|
9
|
+
What Codex starts (MCP servers, tool commands such as `sleep 40`) leads process groups of its own (read from the live table). A group stop alone does not reach them if the native is SIGKILLed before it cleans up, so the hub's stop records them by parent links, before the stop and every 100 ms of the grace period, then freezes what is left (SIGSTOP) before reading the table again and killing it, and is done only when the table shows none of it. Tests (`test/shutdown.test.ts`): a launcher that waits for a child ignoring SIGTERM, a member that outlives its leader, a descendant in a group of its own, a process started and orphaned during the grace period (the test fails without the recording during the grace period), no process table at all (the group's own answer decides), and a table that keeps showing a member (the stop gives up at its deadline, never claims done).
|
|
10
|
+
|
|
11
|
+
Live interrupted-arm smokes, on case 0 (`pallets_click_task:2068`) with real agents, Codex 0.160.0 and Claude Code 2.1.288. Each prepared its own run directory and built a protect list of every earlier benchmark artifact and native session file. After each run exited, the process table was searched for anything with a fixture in its argv and `lsof` for anything with a fixture as its working directory.
|
|
12
|
+
|
|
13
|
+
| Run | Source | Arm | Stopped at | Signal | Cleanup | Left running (argv, cwd) | Restoration |
|
|
14
|
+
|---|---|---|---|---|---|---|---|
|
|
15
|
+
| A | `9d9f552` | solo Codex | 30.5 s into the work, Codex mid-turn | SIGTERM to the runner | `clean` (normal shutdown 533 ms) | nothing, nothing | inputs, sibling locks |
|
|
16
|
+
| B | `9d9f552` | advisory | 41.0 s into the work | SIGINT to the runner's process group | `clean` (918 ms) | nothing, nothing | inputs, locks, trust |
|
|
17
|
+
| C1 | `9d9f552` | solo Claude | completed normally (44.3 s) | none | `clean` (460 ms) | nothing, nothing | inputs, locks, trust |
|
|
18
|
+
| C2 | `9d9f552` | advisory (the next arm of C1's run) | 8.9 s into the work | SIGTERM to the runner | `clean` (707 ms) | nothing, nothing | inputs, locks, trust |
|
|
19
|
+
| D | `9d9f552` | advisory | setup, before the hub started (`ahub kill` said "hub is not running", recorded) | SIGTERM to the runner | `clean` (170 ms) | nothing, nothing | inputs, locks |
|
|
20
|
+
| E | `9d9f552` | turn-free | 31.1 s into the work, and again during teardown | SIGINT to the group, twice | `clean` (916 ms) | nothing, nothing | inputs, locks, trust |
|
|
21
|
+
| S1 | `9d9f552` | advisory | native startup: the hub up, Claude's channel prompt confirmed, its sandbox probe not run yet | SIGTERM to the runner | `clean` (380 ms) | nothing, nothing | inputs, locks, trust |
|
|
22
|
+
| S2 | `9d9f552` | advisory | native readiness: Claude's probe done, Codex's thread started and its MCP ready, the Codex probe not finished | SIGINT to the group | `clean` (561 ms) | nothing, nothing | inputs, locks, trust |
|
|
23
|
+
| V | `9d9f552` | solo Codex, with a visitor: a `sleep` started in the arm's fixture by the test, not by the arm | 30.4 s into the work | SIGTERM to the runner | `incomplete_or_unknown`: the visitor, working in the fixture, unresolved and left alone | the visitor only | withheld; `restore.ts` refused while the visitor ran, restored once it left |
|
|
24
|
+
| F | `c9d7ff0` | advisory | 40.2 s into the work | SIGINT to the group | `incomplete_or_unknown`: the Orca terminal shell working in the fixture was not recorded | nothing, nothing | withheld; `restore.ts` restored after its checks |
|
|
25
|
+
|
|
26
|
+
Runs A, B, E, V and S2 were repeated on `5a30876`, the last commit to change code, with the same outcomes (every cleanup `clean`, V `incomplete_or_unknown` and recovered once its visitor left, nothing of a fixture left running). V and F are the fail-closed path. In V, at the head's source, a process the arm did not start sits in the fixture: the runner leaves it alone, keeps the protected inputs and sibling locks in place, writes the record to `recovery/` and stops the cohort; `bun scripts/benchmarks/restore.ts --run <run>` refuses while the visitor runs (it names it twice: unresolved in the record, and working in the fixture), and once it is gone restores the modes, moves the record into `runs/` and leaves the upstream checkout readable again. In F, on an earlier source, the same path was taken for real: the Orca terminal's login shell, which runs Claude's launcher and works in the fixture, outlives the terminal close by a moment and was not recorded; since `d692534` it is recorded as part of Claude's launch chain (the parent of the launcher, working in the fixture). Setup-stage stops are recorded as `interrupted` with their error kept. C1's completed arm waited for Claude's turn end (bound 30 s): the session's `turn_duration` row (proved on the probe turn in every Claude arm that got that far) arrived after 1.01 s (on `1b33179` it had already been written when the wait began), and its tree was the same after the verified teardown as at the end of its active time. Every arm that reached Codex recorded `conditions.codex.skills` from `skills/list`, a summary only (counts by scope and a hash of the names): 177 user skills, all enabled, in every arm, and a number of system skills that Codex reported differently from arm to arm on the same machine and Codex configuration: 4 or 5 within the batch on `9d9f552`, 4 on `1b33179`, none on `65d6acc` and 5 on `24390de`, while direct `skills/list` calls in between saw 5. The cause is in Codex and was not established. Each record therefore says what `skills/list` answered for its arm at thread start, which a comparison of arms can read; that this was the session's effective skill availability is not shown. No record keeps the `skills/list` answer itself (no skill path in any record since `24390de`, against 183 paths in a record from the source before). D stopped before Codex and says so. Process start times in the records are read in UTC since `65d6acc`. Runs on the sources in between (`ab11387`, `488a2df`, `d692534`, `557c40c`, `28475c3`, `52972ff`, `24390de`, `65d6acc`, `3d017ed`, `1b33179`) gave the same outcomes; those before `c9d7ff0` did not yet scan working directories.
|
|
27
|
+
|
|
28
|
+
The turn-end marker: in the 0.12.4 release runs, all 126 Claude turns that ended with an `end_turn` answer were followed by a `turn_duration` row of the session before the next activity; the 5 transcripts whose last activity has no such row are cut-off attempts.
|
|
29
|
+
|
|
30
|
+
Fault fixtures (`test/benchmarks/teardown.test.ts`, in-memory process tables and one real process tree): a normal shutdown is `clean` with nothing signalled; a lost acknowledgement gets a fallback on proven identities only, group leaders by their group: `clean_with_fallback`; a fallback that cannot stop an actor, an unreadable table and unreadable working directories are `incomplete_or_unknown`; a first read that fails keeps every recorded actor for the fallback; a reused pid, a replacement hub on the same fixture and a foreign process naming the fixture are never signalled, and the last two leave the cleanup incomplete; Codex's argv without the fixture is owned by lineage; a background job in a recorded group is the arm's while that group is known, and one that left it, or a shell merely working in the fixture, is left alone and keeps the cleanup open. Turn end: tool-use-only rows, a final answer without the marker, another session's marker and a new turn after the marker do not pass; the bounded wait reports ended, timeout, interrupted and unreadable. Restoration: modes back parents first with failures named, a trust entry the user changed kept, and the recovery tool refusing while the runner, a recorded process or anything working in the fixture runs, or while working directories cannot be read, then restoring and moving the kept record into `runs/`. The ledger keeps the frozen transcript prefix when Claude Code appends after it, reports `late_append_bytes`, and leaves an open turn's settlement unknown.
|
|
31
|
+
|
|
32
|
+
## #112 and #109
|
|
33
|
+
|
|
34
|
+
Unit tests: the seen-cap and touched-limit notices survive a dropped offer, are spent by an acknowledgement, and a new session starts both and its touched list afresh (the session test fails with the reset removed); a rewrite with HEAD's bytes under a named directory is neither named nor counted against `current()`, a real change is; a names-only offer counts its names in `named`, and the ledger reports them per path (unknown for events written before 0.12.5). Split records: a hand-over is tagged with the new owner's profile, observations count only with the current profile, an unknown version makes the prediction unknown, and a `routing` record is made only for a first assignment routing chose while the overlapped task has not started (the reassignment case fails with that rule removed); Codex's version comes from app-server's `userAgent`, Claude Code's from its transcript (daemon test). Calibration and the held-out assessment (T3) and opt-in rerouting (T4) stay open: the data source and sample size are preregistered in #110 first.
|
|
35
|
+
|
|
36
|
+
## Upgrade
|
|
37
|
+
|
|
38
|
+
`scripts/smoke-recovery-09-10.ts` recovered the real published 0.12.4 package (protocol 13) into the candidate (protocol 13): operation `46160c3c-0f87-4be3-8cfa-3739f36bcab7`, queued envelope ids and the task identity and state digest preserved.
|
|
39
|
+
|
|
40
|
+
## Automated gate and review
|
|
41
|
+
|
|
42
|
+
`scripts/check.sh` at `5a30876`: 730 tests passed, 0 failed, 3,677 expectations across 66 files; typecheck, bundle freshness, npm package contents and process ownership passed; `check: OK`. CI on `5a30876`, `241fbe7`, `9d9f552`, `c988054`, `1b33179`, `3d017ed` and `fcc27e9`: both platforms passed at the first attempt. CI on `65d6acc`: Ubuntu passed; macOS passed on its third attempt. The first hung in `test/daemon.test.ts` after `conflicts: with experiments.stale_notices deliver ...` until the job's 15-minute limit, with no test reaching its own timeout (the event loop was blocked, as in one local run earlier in this work that showed the test process at 99.6% CPU with no child process). The second failed `a peer can never answer a permission request` (`permission=cancelled`), a test that failed the same way on this branch before and on the 0.12.4 branch. Neither reproduced locally: 8 runs of `daemon.test.ts` alone and 8 of it with the facts, benchmark and events tests all passed. Both are recorded as open, not as fixed.
|
|
43
|
+
|
|
44
|
+
Independent host Claude Code OCR delegation reviews (read-only reviewers with the OCR rule groups, `REVIEW.md`, the `AGENTS.md` invariants and the issue contracts) on the exact head. The first round, on `ce1901d`, found that #113 had been amended ("Review correction") before the first commit and that the runner met only its earlier contract; the runner's teardown was rebuilt to the amended contract. Its other findings, all fixed: a localized `ps` that parsed to nothing and still read as clean; restoration while a survivor could read the inputs; the completion wait running before the agents were paused; a daemon still finishing its exit counted as a leftover; fallback by pid where a group was needed; a transcript read error able to skip the teardown; process groups of Codex's own children; group stops confirmed by the leader's exit alone; a new session told about the old one's touched files; the touched-limit notice without a name cap; the HEAD check reading a second version of a file; Claude's version lost behind a big transcript row; escalations and relays counted as routing records; `route explain` and the record describing different pairs; the skills condition a constant; `teardown.ts` unpinned; ledger gaps; the 0.12.4 upgrade claim without proof. The second round, on `615dce9`, found a macOS EPERM from a group whose members were all zombies (CI), recorded actors dropped when the first table read failed, a background job that could escape the reads, `restore.ts` restoring without a recovery record or while the runner ran, a replacement daemon adoptable at teardown, kept records invisible to grading, records with teardown errors graded, an end reason overwritten by later flags, `restoration.json` silent about sibling locks, the completion wait writing the agent's index, failures of an escalation or decline charged to the new owner's split observations, and smaller items; all were fixed, and the live runs above were repeated with the working-directory scan. The third round, on `049b2cd`, found that a process the tree started during the stop's grace period escaped it (the third gap of the same kind, so the stop was redesigned: freeze what is left, read the table again, then kill; the rule is in `AGENTS.md`), split observations that charged nobody for a decline or escalation away, large files outside the HEAD comparison, a run-level restoration that an early exception could let through, unbounded runner commands and an unhandled SIGHUP, a completion-wait hash that followed links, a registration left by an interrupted setup, `restore.ts` not taking back the trust entry and `runner.py restore` as a second, unchecked path, a doc overclaim and inconsistencies in this record; all were fixed and the live runs above repeated. The fourth round, on `a92b475`, found the benchmark fallback still reading the table while the actors ran (it now records what they start while it waits and freezes before SIGKILL), the ledger counting attempts grading had made unavailable (one gate now serves both), a completion wait able to run past the active limit or let a write into the patch unflagged, trust restored while containment was uncertain and never retried after a failure, a recovery that followed links, a ledger written in place, an unbounded `lsof`, an unrecorded member of a group whose id may have been reused (left alone, and the stop no longer called done), descendants without a grace period, a synchronous table read blocking the hub, and PII-named files dropped from the counts; all were fixed and the live runs above repeated, the fail-closed path (V) included. The fifth round, on `a390aec`, found a startup race in a new stop test that turned Linux CI red (its children now report readiness only after their handler is set), a HEAD check that could follow a link or block on a fifo swapped in for a large file (now hashed in-process, opened as facts open files), a narrow window that could leave a process frozen and not killed (what is frozen is now killed or continued, in the hub's stop and the benchmark fallback alike), group membership judged from a recorded instead of a current pgid, end flags that grading and the ledger did not report (now `end_story`), a tree check that ended before the kill (now from the end of the active time to after the verified teardown), the recovery's ledger written in place, the regime in force missing from split profiles, and smaller items; all were fixed and the live runs above repeated. The sixth round, on `768b60e`, found the `skills/list` answer, with absolute skill paths, kept in every run record beside its hashed summary (its response is no longer recorded), records from before 0.12.3 graded invalid by the ledger instead of unknown, the capture of an arm's processes and the new pins without tests (the capture is now a function tested on fixture tables, the pins are tested in `prepare` and `grade`), a stop that read as done when a launcher had exited and left members in its group (it now fails, its group unsignalled), pipes not dropped after a successful stop, an offer read back late spending the notice of a later drop of the same file, large files hashed again at every boundary (now once per version), and smaller items; all were fixed. The seventh round, on `24390de`, found no Critical or Important defect; its Minor findings were fixed: process start times read in local time, so a reader in another time zone (or a zone change during a run) would see every identity as changed (now read in UTC); descendants leading groups of their own given no SIGTERM before the freeze (a tool command such as `git` killed without a chance to remove its lock; now each gets its own SIGTERM and the grace period); a pre-0.12.3 record with a definite isolation failure reported as unknown; the restoration ledger rewritten every 5 s in full (now only when the recorded actors change, without restored sibling maps); a recovery that could reach through a replaced parent directory; and test gaps (the adapter's launcher-and-native group stop, a reused pid in the hub's stop). The eighth round, on `fcc27e9`, again found no Critical or Important defect; its Minor findings were fixed: a process that a read after the freeze showed for the first time was killed without being frozen (it is now frozen and the table read again, up to five times, in the hub's stop and the runner's fallback alike); after a failed read following the freeze, a pid whose STOP had failed could still be killed from the earlier read (now only what the STOP reached is killed); a frozen tree could wait on a slow `ps` for its SIGKILL (the read after the freeze is given up after 1 s); a launcher that had exited, whose pid now led someone else's group, failed every later stop and so blocked a Codex restart (a live process holding the leader's pid now proves the group is not the hub's, and zombies are not counted); large-file hashing in one call is capped at 64 MiB; the completion wait's bound is taken when the wait starts; a `skills/list` error is recorded as its class, not its message; a process not the arm's is recorded by program name, never its arguments; the tree hash and the metadata hash open files without blocking; and `restoration.json` and the recovered records are written atomically. The ninth round, on `3d017ed`, found no Critical or Important defect; its Minor findings were fixed: a per-call budget for large-file hashing that let an integration target move without a byte changing (whether large files are compared now depends on their total size alone, and a file is hashed only as the version observed), a calibration record for work routing gave back to its proposer, whose observation is left out (no record now either), a process frozen in a later round that was continued instead of killed when the next read failed (now killed by the last read that showed it), 0.12.3 and 0.12.4 records' `cleanup_complete` read as a verified teardown by the ledger (unknown now), and `setupMs` that counted the teardown of an arm stopped in setup. The tenth round, on `c988054`, found one Important defect, fixed: routing-time calibration records would almost always read `not available`, because the owner of the overlapped task is busy taking it when the next task is routed (it now counts as available while that task is not started, as the routed peer does; #109 amended). Its Minor findings were fixed too: an exited launcher's group id that could be reused before a later stop (the group is now followed after the launcher exits until it is gone), a group SIGTERM sent after the leader may have been reaped, no capture during the completion wait (now every 5 s there too), the tree hashed after the pause rather than first, pre-0.12.5 records re-judged by the ledger (now judged by their own flags, their teardown shown as not verified), and documentation of what `teardown_errors` covers. Live runs S1 and S2 now cover a stop during native startup and readiness. The eleventh round, on `241fbe7`, found no Critical or Important defect; its Minor findings were fixed: a process that led a group of its own only after it was recorded had that group's other members left out of what the runner owns, which could read as a complete cleanup (groups are now followed by the table as it is now), what the runner froze and did not kill was continued only when one more read succeeded (now by the last read that showed it), the failed stop of an exited launcher did not name the members it left (now it does), a routing record written after the board write without a guard (now guarded), and documentation of the Linux clock-step limit, the exited launcher's limit and what 0.12.3 and 0.12.4 recorded. The twelfth round, on `f63c5ea`, found no Critical or Important defect. Two of its Minor findings were documentation and were fixed: the 64 MiB rule described as a limit per file, where it is the total of the large files in one comparison, and the skills condition described as what each arm had, where it is what `skills/list` answered at thread start. Its other Minor findings were deferred. From the seventh round on, the round summaries above name the findings that were fixed; the Minor findings they do not name were recorded as known limits (below) or deferred. The pull request lists the deferred ones, and later rounds.
|
|
45
|
+
|
|
46
|
+
Known limits: the hub sees its own hooks only, so a profile does not tell the user's or plugins' hooks apart, and only Claude and Codex report a version; a process that leaves the recorded parent links and groups between two reads (every 5 s while the agents work) is not recorded, and keeps the cleanup open only if it names a fixture or works in one; the hub's own stop records what the tree starts during the grace period every 100 ms, so a process started and orphaned within one such interval escapes it (an unrecorded process left in a recorded group is never signalled, and the stop is then not called done); identity is the pid with a start time of one-second resolution, and on Linux `ps` derives that time from the boot time, which moves when the clock is stepped, so a stop across a clock step sees its processes as replaced and leaves them alone: the hub's own stop then reads incomplete, never done, while the runner's teardown counts such an owned process as gone and keeps the cleanup open only if it names or works in the fixture (the benchmark runs on macOS); the hub's own stop does not sweep the group of a Codex launcher that was killed from outside before the stop (while that group has members the stop fails instead of reading as done, naming them), nor processes that left that group for groups of their own before the stop (an MCP server or a tool command orphaned with the launcher): its pipes are dropped so they cannot keep the hub alive; a file too large to read is compared by its raw bytes, so with git's end-of-line conversion or LFS it reads as changed and is named; the trust entry is written by read, change and rename, so a write another Claude Code session makes to `~/.claude.json` in the milliseconds between is lost (as before this release).
|
package/package.json
CHANGED