crewbus 10.0.11 → 10.0.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/docs/DELIVERY.md CHANGED
@@ -1,210 +1,210 @@
1
- # Delivery semantics
2
-
3
- How the bus behaves when things go wrong. Read this before building
4
- orchestrators on top of `send`/`spawn`.
5
-
6
- ## At-least-once, not at-most-once
7
-
8
- - **Files are the truth.** `send` writes one immutable file per DM
9
- (`dm/<recipient>/<id>.json`) or one `broadcast/<batch>.json` for
10
- fan-outs over 20. A send that echoes an id **landed**.
11
- - **Push is at-least-once.** The opencode watcher and the hook helpers
12
- claim `delivered/<agent>/<msg>.json` markers with an exclusive create,
13
- and the cursor advances only on successful injection. A failed push
14
- (e.g. into a stale session) releases the marker and is **retried**, so a
15
- recipient may see a DM twice across a crash — never zero times.
16
- - **Pull (`inbox`) is idempotent.** No mark-read side effects; page with
17
- `--after`, filter with `--unacked`. `redeliver --id|--all` rewinds the
18
- cursor after a dead watcher consumed mail.
19
-
20
- ## Ordering
21
-
22
- - **No global order.** DMs from different senders interleave arbitrarily.
23
- - **Per-sender order is wall-clock + id.** `inbox`/`gather`/`thread`
24
- present oldest first by `at` timestamp; ids sort ties (4 random bytes,
25
- unique under parallel sends).
26
- - **Channels append.** `channels/<name>.log.jsonl` is append-only; concurrent
27
- posts serialize on file append. Sync merges channel logs **union-by-id**,
28
- so both sides' lines survive a partition.
29
-
30
- ## What happens when a worker dies mid-brief
31
-
32
- - The **brief survives** — it is a file, not a process. A replacement
33
- worker with the same name sees it via `inbox` (stale names are reusable;
34
- spawning refuses names with a *live* worker behind them).
35
- - A **detached worker's reply may never come.** `spawn-status` shows
36
- `running? reply landed? acked?` plus a log tail: an exited worker with
37
- no reply failed silently — read its log in `.crewbus/logs/`.
38
- - **Pool mode** (`pool --count N --pool-size S`) replaces dead workers
39
- automatically up to N total, with backpressure (queue refuses past
40
- 4× pool size).
41
-
42
- ## Presence you can trust (recycled-pid guard)
43
-
44
- `spawn-status` liveness used to be `kill(pid, 0)` alone: after a reboot
45
- the OS recycles pids and a dead worker could read as "running". Status
46
- now also resolves the process start time (`ps` elapsed on unix,
47
- `Get-Process` on Windows, 60s memo) and compares it against the recorded
48
- `spawnedAt` (±120s skew): a live pid whose process started *after* the
49
- spawn is a stranger, and `alive` flips to false. The verdict rides along
50
- as `aliveVerified` (`true` corroborated, `false` contradicted, `null`
51
- unverifiable on this platform — the old kill-0 verdict stands). The
52
- override needs positive evidence (start time after spawn + skew), so a
53
- false "dead" takes a >2min clock jump between spawn and check — rare,
54
- and missing data never flips a verdict. Practical effect: after a
55
- reboot, recycled pids read dead instead of running, so `respawn` is
56
- allowed instead of wrongly refused. Reconcile sweeps (`doctor`,
57
- post-reboot `spawn-status --all` review) remain the operator's job for
58
- now.
59
-
60
- ## Progress checkpoints
61
-
62
- Long briefs die mid-way; re-deriving progress from files wastes the
63
- resume. Workers checkpoint every few steps (or before anything risky):
64
-
65
- ```sh
66
- crewbus send --from <you> --to <lead> --reply <brief-id> --checkpoint --body "done X / next Y"
67
- ```
68
-
69
- (`dm_send` takes `checkpoint: true`; same on the opencode `dm-send`
70
- tool.) Checkpoints ride the normal DM thread, so `thread`/`gather`,
71
- sync, and push delivery all work unchanged — but they are *progress*,
72
- not work awaiting acceptance:
73
-
74
- - `--unacked` (CLI and `dm_inbox`) and the `ack --timeout` hint skip
75
- them; transcripts label them (`checkpoint: progress`, `[checkpoint]`
76
- in digests).
77
- - `spawn-status` reply detection ignores them: only a non-checkpoint
78
- reply to the brief counts as "reply landed".
79
- - `ack --all` still records them if run (harmless); rate limits and
80
- duplicate suppression apply like any send, so checkpoint every few
81
- steps — not every tool call.
82
-
83
- A respawned worker reads the latest checkpoint off the thread
84
- (`inbox`/`thread --id <brief-id>`) and continues from "next Y" instead
85
- of restarting. Checkpoints are unverified progress: they never feed
86
- `result record` / `race` winner logic (that still needs `ack --verify`).
87
-
88
- ## Pool supervision + re-attach
89
-
90
- `pool` briefs all workers up front, boots at most `--pool-size`
91
- concurrently, and replaces exits until N total — but the supervision loop
92
- runs inside the invoking CLI process. If that process dies (lead restart,
93
- shutdown, supervision-window timeout), `pool-resume --id <pool> --from
94
- <you>` re-attaches: it reconciles every launched worker (replied → done,
95
- live → re-adopted, dead w/o reply → done with a `respawn` hint), then
96
- supervises the unstarted remainder for a fresh window. Briefs are reused
97
- by id, never duplicated; one supervisor at a time via an advisory pool
98
- lock (TTL expiry lets a later attach take over after a crash).
99
- `finishedAt` marks true completion only — a timed-out pool stays
100
- resumable. Pools created before the resumable schema (no queue cursor)
101
- are refused with guidance. Lead/admin only.
102
-
103
- Windows note: console-less detached `cmd.exe` wrappers linger up to ~60s
104
- after their payload exits, so natural-exit detection on `shell:true`
105
- harnesses (opencode/claude/codex/cursor/generic) trails by up to a
106
- minute; kills report immediately. Pool turnover and `spawn-status`
107
- inherit the tail — size supervision windows accordingly.
108
-
109
- ## Long runs (the 10-day task)
110
-
111
- Don't run one agent for 10 days — run a board for 10 days. Full playbook:
112
- `docs/LONG_RUNS.md` (setup checklist, wave pattern, recovery ladder).
113
- The short version: sessions are a cache, the board is the database —
114
- rotate workers in waves, checkpoint progress, poll digest-first, persist
115
- tokens to files, bound everything, and let compaction/restart/reboot
116
- happen. A cold session plus a warm board is a crew that never sleeps.
117
-
118
- ## Unacknowledged-brief timeout (retry / reassign)
119
-
120
- `ack` is orthogonal to delivery: a reply that sits `--unacked` is work
121
- nobody accepted. The reassign pattern:
122
-
123
- ```sh
124
- # what is still open — and how long has it waited?
125
- crewbus inbox --from lead --unacked --older-than 10m
126
-
127
- # the brief is a file: re-brief a fresh worker on the same thread
128
- crewbus send --from lead --to w9 --reply <brief-id> --body "picking this up (w3 went silent)"
129
-
130
- # the dead watcher's mail, if any, can be replayed
131
- crewbus redeliver --from w3 --all
132
- ```
133
-
134
- There is no automatic reaper: leads own the timeout policy for their
135
- crew. `spawn --timeout 10m` records `deadlineAt` (`spawn-status` flags
136
- passed deadlines — reap with `spawn-kill`/`stop`); `stop --all --from <you>`
137
- terminates everything when the whole run is lost.
138
-
139
- ## Sync partitions
140
-
141
- - Message files are immutable with unique ids: sync is **conflict-free
142
- union** — both sides end up with both sides' mail.
143
- - Presence, cursors and groups merge by **hybrid logical clock +
144
- version counter** (wall-mtime fallback for legacy docs); newer
145
- `(hlc, v)` wins, ties keep local content and converge clocks so the
146
- next round is `pulled 0, pushed 0`.
147
- - **Prune stays local.** Deletions replicate only as **tombstones**
148
- (`tombstones/*.json`, synced): a pruned message never resurrects on a
149
- partitioned peer, but the peer's other history is untouched.
150
- - Rounds after the first are incremental (per-peer cursor, 60s overlap);
151
- `--dry-run` previews pulls/pushes/tombstones without touching files.
152
-
153
- ## Worker lifetime
154
-
155
- - `spawn` defaults to **oneshot**: headless workers (`claude -p`,
156
- `codex exec`, `cursor-agent -p`, …) run the brief and exit. Their mail
157
- **waits for pull** — a finished worker cannot receive push.
158
- - `--persistent` marks long-lived peers (opencode sessions, daemons)
159
- that stay addressable. `spawn-status` shows the recorded lifetime.
160
- - Either way: **finished workers' DMs are files** — `inbox --from <name>`
161
- reads them after the process is gone.
162
- - **Session-id capture (respawn prerequisite).** Spawn boots the four
163
- JSON-output harnesses with machine-readable logs (`opencode run
164
- --format json`, `claude -p --output-format json`, `grok
165
- --output-format json`, `codex exec --json`) so the harness session id
166
- can be recovered from the worker log. `spawn-status` lazily extracts it
167
- into `worker-sessions/<name>.json` (kept beside — not inside — the
168
- agent doc, so presence heartbeats can never wipe it; machine-local,
169
- never synced). Provenance is tracked (`preassigned` /
170
- `preassigned-confirmed` / `log` / `log-override` — a disagreeing log
171
- wins, in case the installed harness ignored the flag).
172
- - **Pre-assigned ids (the mid-run-kill case).** Log parsing can't cover a
173
- worker killed before emitting anything (notably grok, whose id only
174
- arrives end-of-run). Where the CLI supports creating with an id,
175
- `bootWorker` mints a UUID and passes it in (`claude --session-id`,
176
- grok `-s`), so the binding holds a resumable id from boot. opencode
177
- has no pre-set flag (its id arrives in the first stream event — the
178
- residual window is milliseconds); codex relies on `thread.started`;
179
- cursor/agy offer neither.
180
- - **Respawn (`respawn --from <lead> --to <worker>`).** Reboots one dead
181
- worker in its *same* harness conversation: `opencode run --session`,
182
- `claude -p --resume` (+ `CLAUDE_CODE_RESUME_INTERRUPTED_TURN=1` so a
183
- killed turn continues), `codex exec resume`, `grok -r`, agy
184
- `--conversation`, cursor `--resume`; generic has no continuity and
185
- re-boots fresh. The catch-up brief points at the original prompt file
186
- and threads the same reply id, so the worker continues instead of
187
- restarting; codex points at the catch-up file itself (its positional
188
- prompt can't attach files). Refuses live workers unless `--force`
189
- (kills first, own-crew scoped like `spawn-kill`); refuses unknown,
190
- never-captured, and pre-capture-era workers with re-brief guidance.
191
- Attempts count on the binding; `--dry-run` previews. Lead/admin only.
192
-
193
- ## Poll discipline (long runs)
194
-
195
- Plain `inbox` dumps everything — over days that burns context.
196
- Poll digest-first instead, escalate only on hits:
197
-
198
- 1. `crewbus inbox --from <you> --unacked --digest` — cheap
199
- one-line-per-message sweep (`dm_inbox({ agent, unacked: true,
200
- digest: true })` on MCP).
201
- 2. Filtered full read — same sweep narrowed before expanding:
202
- `inbox --from <you> --unacked --grep <pat>` /
203
- `--priority high` (MCP: `grep` / `priority` params).
204
- 3. `thread --id <msg-id>` / `gather` for context — only once a hit
205
- proves worth reading.
206
-
207
- Cadence: poll cheap and often; full reads on signal. Push wakes you;
208
- digest tells you if it matters. Checkpoints compose: digests mark
209
- progress notes `[checkpoint]` (skipped by `--unacked` triage, never
210
- need ack), so the cheap poll still shows where a thread stands.
1
+ # Delivery semantics
2
+
3
+ How the bus behaves when things go wrong. Read this before building
4
+ orchestrators on top of `send`/`spawn`.
5
+
6
+ ## At-least-once, not at-most-once
7
+
8
+ - **Files are the truth.** `send` writes one immutable file per DM
9
+ (`dm/<recipient>/<id>.json`) or one `broadcast/<batch>.json` for
10
+ fan-outs over 20. A send that echoes an id **landed**.
11
+ - **Push is at-least-once.** The opencode watcher and the hook helpers
12
+ claim `delivered/<agent>/<msg>.json` markers with an exclusive create,
13
+ and the cursor advances only on successful injection. A failed push
14
+ (e.g. into a stale session) releases the marker and is **retried**, so a
15
+ recipient may see a DM twice across a crash — never zero times.
16
+ - **Pull (`inbox`) is idempotent.** No mark-read side effects; page with
17
+ `--after`, filter with `--unacked`. `redeliver --id|--all` rewinds the
18
+ cursor after a dead watcher consumed mail.
19
+
20
+ ## Ordering
21
+
22
+ - **No global order.** DMs from different senders interleave arbitrarily.
23
+ - **Per-sender order is wall-clock + id.** `inbox`/`gather`/`thread`
24
+ present oldest first by `at` timestamp; ids sort ties (4 random bytes,
25
+ unique under parallel sends).
26
+ - **Channels append.** `channels/<name>.log.jsonl` is append-only; concurrent
27
+ posts serialize on file append. Sync merges channel logs **union-by-id**,
28
+ so both sides' lines survive a partition.
29
+
30
+ ## What happens when a worker dies mid-brief
31
+
32
+ - The **brief survives** — it is a file, not a process. A replacement
33
+ worker with the same name sees it via `inbox` (stale names are reusable;
34
+ spawning refuses names with a *live* worker behind them).
35
+ - A **detached worker's reply may never come.** `spawn-status` shows
36
+ `running? reply landed? acked?` plus a log tail: an exited worker with
37
+ no reply failed silently — read its log in `.crewbus/logs/`.
38
+ - **Pool mode** (`pool --count N --pool-size S`) replaces dead workers
39
+ automatically up to N total, with backpressure (queue refuses past
40
+ 4× pool size).
41
+
42
+ ## Presence you can trust (recycled-pid guard)
43
+
44
+ `spawn-status` liveness used to be `kill(pid, 0)` alone: after a reboot
45
+ the OS recycles pids and a dead worker could read as "running". Status
46
+ now also resolves the process start time (`ps` elapsed on unix,
47
+ `Get-Process` on Windows, 60s memo) and compares it against the recorded
48
+ `spawnedAt` (±120s skew): a live pid whose process started *after* the
49
+ spawn is a stranger, and `alive` flips to false. The verdict rides along
50
+ as `aliveVerified` (`true` corroborated, `false` contradicted, `null`
51
+ unverifiable on this platform — the old kill-0 verdict stands). The
52
+ override needs positive evidence (start time after spawn + skew), so a
53
+ false "dead" takes a >2min clock jump between spawn and check — rare,
54
+ and missing data never flips a verdict. Practical effect: after a
55
+ reboot, recycled pids read dead instead of running, so `respawn` is
56
+ allowed instead of wrongly refused. Reconcile sweeps (`doctor`,
57
+ post-reboot `spawn-status --all` review) remain the operator's job for
58
+ now.
59
+
60
+ ## Progress checkpoints
61
+
62
+ Long briefs die mid-way; re-deriving progress from files wastes the
63
+ resume. Workers checkpoint every few steps (or before anything risky):
64
+
65
+ ```sh
66
+ crewbus send --from <you> --to <lead> --reply <brief-id> --checkpoint --body "done X / next Y"
67
+ ```
68
+
69
+ (`dm_send` takes `checkpoint: true`; same on the opencode `dm-send`
70
+ tool.) Checkpoints ride the normal DM thread, so `thread`/`gather`,
71
+ sync, and push delivery all work unchanged — but they are *progress*,
72
+ not work awaiting acceptance:
73
+
74
+ - `--unacked` (CLI and `dm_inbox`) and the `ack --timeout` hint skip
75
+ them; transcripts label them (`checkpoint: progress`, `[checkpoint]`
76
+ in digests).
77
+ - `spawn-status` reply detection ignores them: only a non-checkpoint
78
+ reply to the brief counts as "reply landed".
79
+ - `ack --all` still records them if run (harmless); rate limits and
80
+ duplicate suppression apply like any send, so checkpoint every few
81
+ steps — not every tool call.
82
+
83
+ A respawned worker reads the latest checkpoint off the thread
84
+ (`inbox`/`thread --id <brief-id>`) and continues from "next Y" instead
85
+ of restarting. Checkpoints are unverified progress: they never feed
86
+ `result record` / `race` winner logic (that still needs `ack --verify`).
87
+
88
+ ## Pool supervision + re-attach
89
+
90
+ `pool` briefs all workers up front, boots at most `--pool-size`
91
+ concurrently, and replaces exits until N total — but the supervision loop
92
+ runs inside the invoking CLI process. If that process dies (lead restart,
93
+ shutdown, supervision-window timeout), `pool-resume --id <pool> --from
94
+ <you>` re-attaches: it reconciles every launched worker (replied → done,
95
+ live → re-adopted, dead w/o reply → done with a `respawn` hint), then
96
+ supervises the unstarted remainder for a fresh window. Briefs are reused
97
+ by id, never duplicated; one supervisor at a time via an advisory pool
98
+ lock (TTL expiry lets a later attach take over after a crash).
99
+ `finishedAt` marks true completion only — a timed-out pool stays
100
+ resumable. Pools created before the resumable schema (no queue cursor)
101
+ are refused with guidance. Lead/admin only.
102
+
103
+ Windows note: console-less detached `cmd.exe` wrappers linger up to ~60s
104
+ after their payload exits, so natural-exit detection on `shell:true`
105
+ harnesses (opencode/claude/codex/cursor) trails by up to a
106
+ minute; kills report immediately. Pool turnover and `spawn-status`
107
+ inherit the tail — size supervision windows accordingly.
108
+
109
+ ## Long runs (the 10-day task)
110
+
111
+ Don't run one agent for 10 days — run a board for 10 days. Full playbook:
112
+ `docs/LONG_RUNS.md` (setup checklist, wave pattern, recovery ladder).
113
+ The short version: sessions are a cache, the board is the database —
114
+ rotate workers in waves, checkpoint progress, poll digest-first, persist
115
+ tokens to files, bound everything, and let compaction/restart/reboot
116
+ happen. A cold session plus a warm board is a crew that never sleeps.
117
+
118
+ ## Unacknowledged-brief timeout (retry / reassign)
119
+
120
+ `ack` is orthogonal to delivery: a reply that sits `--unacked` is work
121
+ nobody accepted. The reassign pattern:
122
+
123
+ ```sh
124
+ # what is still open — and how long has it waited?
125
+ crewbus inbox --from lead --unacked --older-than 10m
126
+
127
+ # the brief is a file: re-brief a fresh worker on the same thread
128
+ crewbus send --from lead --to w9 --reply <brief-id> --body "picking this up (w3 went silent)"
129
+
130
+ # the dead watcher's mail, if any, can be replayed
131
+ crewbus redeliver --from w3 --all
132
+ ```
133
+
134
+ There is no automatic reaper: leads own the timeout policy for their
135
+ crew. `spawn --timeout 10m` records `deadlineAt` (`spawn-status` flags
136
+ passed deadlines — reap with `spawn-kill`/`stop`); `stop --all --from <you>`
137
+ terminates everything when the whole run is lost.
138
+
139
+ ## Sync partitions
140
+
141
+ - Message files are immutable with unique ids: sync is **conflict-free
142
+ union** — both sides end up with both sides' mail.
143
+ - Presence, cursors and groups merge by **hybrid logical clock +
144
+ version counter** (wall-mtime fallback for legacy docs); newer
145
+ `(hlc, v)` wins, ties keep local content and converge clocks so the
146
+ next round is `pulled 0, pushed 0`.
147
+ - **Prune stays local.** Deletions replicate only as **tombstones**
148
+ (`tombstones/*.json`, synced): a pruned message never resurrects on a
149
+ partitioned peer, but the peer's other history is untouched.
150
+ - Rounds after the first are incremental (per-peer cursor, 60s overlap);
151
+ `--dry-run` previews pulls/pushes/tombstones without touching files.
152
+
153
+ ## Worker lifetime
154
+
155
+ - `spawn` defaults to **oneshot**: headless workers (`claude -p`,
156
+ `codex exec`, `cursor-agent -p`, …) run the brief and exit. Their mail
157
+ **waits for pull** — a finished worker cannot receive push.
158
+ - `--persistent` marks long-lived peers (opencode sessions, daemons)
159
+ that stay addressable. `spawn-status` shows the recorded lifetime.
160
+ - Either way: **finished workers' DMs are files** — `inbox --from <name>`
161
+ reads them after the process is gone.
162
+ - **Session-id capture (respawn prerequisite).** Spawn boots the four
163
+ JSON-output harnesses with machine-readable logs (`opencode run
164
+ --format json`, `claude -p --output-format json`, `grok
165
+ --output-format json`, `codex exec --json`) so the harness session id
166
+ can be recovered from the worker log. `spawn-status` lazily extracts it
167
+ into `worker-sessions/<name>.json` (kept beside — not inside — the
168
+ agent doc, so presence heartbeats can never wipe it; machine-local,
169
+ never synced). Provenance is tracked (`preassigned` /
170
+ `preassigned-confirmed` / `log` / `log-override` — a disagreeing log
171
+ wins, in case the installed harness ignored the flag).
172
+ - **Pre-assigned ids (the mid-run-kill case).** Log parsing can't cover a
173
+ worker killed before emitting anything (notably grok, whose id only
174
+ arrives end-of-run). Where the CLI supports creating with an id,
175
+ `bootWorker` mints a UUID and passes it in (`claude --session-id`,
176
+ grok `-s`), so the binding holds a resumable id from boot. opencode
177
+ has no pre-set flag (its id arrives in the first stream event — the
178
+ residual window is milliseconds); codex relies on `thread.started`;
179
+ cursor/agy offer neither.
180
+ - **Respawn (`respawn --from <lead> --to <worker>`).** Reboots one dead
181
+ worker in its *same* harness conversation: `opencode run --session`,
182
+ `claude -p --resume` (+ `CLAUDE_CODE_RESUME_INTERRUPTED_TURN=1` so a
183
+ killed turn continues), `codex exec resume`, `grok -r`, agy
184
+ `--conversation`, cursor `--resume`; `--fresh` re-boots fresh without
185
+ prior conversation continuity. The catch-up brief points at the original prompt file
186
+ and threads the same reply id, so the worker continues instead of
187
+ restarting; codex points at the catch-up file itself (its positional
188
+ prompt can't attach files). Refuses live workers unless `--force`
189
+ (kills first, own-crew scoped like `spawn-kill`); refuses unknown,
190
+ never-captured, and pre-capture-era workers with re-brief guidance.
191
+ Attempts count on the binding; `--dry-run` previews. Lead/admin only.
192
+
193
+ ## Poll discipline (long runs)
194
+
195
+ Plain `inbox` dumps everything — over days that burns context.
196
+ Poll digest-first instead, escalate only on hits:
197
+
198
+ 1. `crewbus inbox --from <you> --unacked --digest` — cheap
199
+ one-line-per-message sweep (`dm_inbox({ agent, unacked: true,
200
+ digest: true })` on MCP).
201
+ 2. Filtered full read — same sweep narrowed before expanding:
202
+ `inbox --from <you> --unacked --grep <pat>` /
203
+ `--priority high` (MCP: `grep` / `priority` params).
204
+ 3. `thread --id <msg-id>` / `gather` for context — only once a hit
205
+ proves worth reading.
206
+
207
+ Cadence: poll cheap and often; full reads on signal. Push wakes you;
208
+ digest tells you if it matters. Checkpoints compose: digests mark
209
+ progress notes `[checkpoint]` (skipped by `--unacked` triage, never
210
+ need ack), so the cheap poll still shows where a thread stands.