crewbus 10.0.11 → 10.0.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +690 -708
- package/bin/crewbus.js +49 -139
- package/bin/lib/identity.js +3 -25
- package/bin/lib/launch.js +68 -25
- package/bin/lib/qrcode.js +2971 -0
- package/bin/lib/relay.js +2 -13
- package/bin/lib/spawn.js +95 -41
- package/bin/lib/web.js +235 -20
- package/docs/CERT_READINESS.md +120 -120
- package/docs/COMPATIBILITY.md +192 -194
- package/docs/CONTROL_PLANE_SPEC.md +9 -11
- package/docs/CREWS.md +55 -55
- package/docs/DELIVERY.md +210 -210
- package/docs/HA.md +151 -151
- package/docs/LEAD_ASSIGN.md +74 -85
- package/docs/M1_CLI_DRAFT.md +2 -2
- package/docs/QUICKSTART.md +70 -70
- package/docs/SPLIT.md +82 -82
- package/docs/THREAT_MODEL.md +68 -69
- package/docs/VERSION_SKEW.md +57 -57
- package/documentation.html +10 -11
- package/examples/hermes-pairing.md +2 -2
- package/index.html +10 -11
- package/opencode/tools/dm-send.js +3 -12
- package/package.json +50 -50
- package/packages/client-runtime/README.md +40 -40
- package/packages/contracts/harness.json +93 -110
- package/packages/contracts/launch.json +1 -1
- package/docs/DESKTOP_SPIKE.md +0 -223
package/docs/DELIVERY.md
CHANGED
|
@@ -1,210 +1,210 @@
|
|
|
1
|
-
# Delivery semantics
|
|
2
|
-
|
|
3
|
-
How the bus behaves when things go wrong. Read this before building
|
|
4
|
-
orchestrators on top of `send`/`spawn`.
|
|
5
|
-
|
|
6
|
-
## At-least-once, not at-most-once
|
|
7
|
-
|
|
8
|
-
- **Files are the truth.** `send` writes one immutable file per DM
|
|
9
|
-
(`dm/<recipient>/<id>.json`) or one `broadcast/<batch>.json` for
|
|
10
|
-
fan-outs over 20. A send that echoes an id **landed**.
|
|
11
|
-
- **Push is at-least-once.** The opencode watcher and the hook helpers
|
|
12
|
-
claim `delivered/<agent>/<msg>.json` markers with an exclusive create,
|
|
13
|
-
and the cursor advances only on successful injection. A failed push
|
|
14
|
-
(e.g. into a stale session) releases the marker and is **retried**, so a
|
|
15
|
-
recipient may see a DM twice across a crash — never zero times.
|
|
16
|
-
- **Pull (`inbox`) is idempotent.** No mark-read side effects; page with
|
|
17
|
-
`--after`, filter with `--unacked`. `redeliver --id|--all` rewinds the
|
|
18
|
-
cursor after a dead watcher consumed mail.
|
|
19
|
-
|
|
20
|
-
## Ordering
|
|
21
|
-
|
|
22
|
-
- **No global order.** DMs from different senders interleave arbitrarily.
|
|
23
|
-
- **Per-sender order is wall-clock + id.** `inbox`/`gather`/`thread`
|
|
24
|
-
present oldest first by `at` timestamp; ids sort ties (4 random bytes,
|
|
25
|
-
unique under parallel sends).
|
|
26
|
-
- **Channels append.** `channels/<name>.log.jsonl` is append-only; concurrent
|
|
27
|
-
posts serialize on file append. Sync merges channel logs **union-by-id**,
|
|
28
|
-
so both sides' lines survive a partition.
|
|
29
|
-
|
|
30
|
-
## What happens when a worker dies mid-brief
|
|
31
|
-
|
|
32
|
-
- The **brief survives** — it is a file, not a process. A replacement
|
|
33
|
-
worker with the same name sees it via `inbox` (stale names are reusable;
|
|
34
|
-
spawning refuses names with a *live* worker behind them).
|
|
35
|
-
- A **detached worker's reply may never come.** `spawn-status` shows
|
|
36
|
-
`running? reply landed? acked?` plus a log tail: an exited worker with
|
|
37
|
-
no reply failed silently — read its log in `.crewbus/logs/`.
|
|
38
|
-
- **Pool mode** (`pool --count N --pool-size S`) replaces dead workers
|
|
39
|
-
automatically up to N total, with backpressure (queue refuses past
|
|
40
|
-
4× pool size).
|
|
41
|
-
|
|
42
|
-
## Presence you can trust (recycled-pid guard)
|
|
43
|
-
|
|
44
|
-
`spawn-status` liveness used to be `kill(pid, 0)` alone: after a reboot
|
|
45
|
-
the OS recycles pids and a dead worker could read as "running". Status
|
|
46
|
-
now also resolves the process start time (`ps` elapsed on unix,
|
|
47
|
-
`Get-Process` on Windows, 60s memo) and compares it against the recorded
|
|
48
|
-
`spawnedAt` (±120s skew): a live pid whose process started *after* the
|
|
49
|
-
spawn is a stranger, and `alive` flips to false. The verdict rides along
|
|
50
|
-
as `aliveVerified` (`true` corroborated, `false` contradicted, `null`
|
|
51
|
-
unverifiable on this platform — the old kill-0 verdict stands). The
|
|
52
|
-
override needs positive evidence (start time after spawn + skew), so a
|
|
53
|
-
false "dead" takes a >2min clock jump between spawn and check — rare,
|
|
54
|
-
and missing data never flips a verdict. Practical effect: after a
|
|
55
|
-
reboot, recycled pids read dead instead of running, so `respawn` is
|
|
56
|
-
allowed instead of wrongly refused. Reconcile sweeps (`doctor`,
|
|
57
|
-
post-reboot `spawn-status --all` review) remain the operator's job for
|
|
58
|
-
now.
|
|
59
|
-
|
|
60
|
-
## Progress checkpoints
|
|
61
|
-
|
|
62
|
-
Long briefs die mid-way; re-deriving progress from files wastes the
|
|
63
|
-
resume. Workers checkpoint every few steps (or before anything risky):
|
|
64
|
-
|
|
65
|
-
```sh
|
|
66
|
-
crewbus send --from <you> --to <lead> --reply <brief-id> --checkpoint --body "done X / next Y"
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
(`dm_send` takes `checkpoint: true`; same on the opencode `dm-send`
|
|
70
|
-
tool.) Checkpoints ride the normal DM thread, so `thread`/`gather`,
|
|
71
|
-
sync, and push delivery all work unchanged — but they are *progress*,
|
|
72
|
-
not work awaiting acceptance:
|
|
73
|
-
|
|
74
|
-
- `--unacked` (CLI and `dm_inbox`) and the `ack --timeout` hint skip
|
|
75
|
-
them; transcripts label them (`checkpoint: progress`, `[checkpoint]`
|
|
76
|
-
in digests).
|
|
77
|
-
- `spawn-status` reply detection ignores them: only a non-checkpoint
|
|
78
|
-
reply to the brief counts as "reply landed".
|
|
79
|
-
- `ack --all` still records them if run (harmless); rate limits and
|
|
80
|
-
duplicate suppression apply like any send, so checkpoint every few
|
|
81
|
-
steps — not every tool call.
|
|
82
|
-
|
|
83
|
-
A respawned worker reads the latest checkpoint off the thread
|
|
84
|
-
(`inbox`/`thread --id <brief-id>`) and continues from "next Y" instead
|
|
85
|
-
of restarting. Checkpoints are unverified progress: they never feed
|
|
86
|
-
`result record` / `race` winner logic (that still needs `ack --verify`).
|
|
87
|
-
|
|
88
|
-
## Pool supervision + re-attach
|
|
89
|
-
|
|
90
|
-
`pool` briefs all workers up front, boots at most `--pool-size`
|
|
91
|
-
concurrently, and replaces exits until N total — but the supervision loop
|
|
92
|
-
runs inside the invoking CLI process. If that process dies (lead restart,
|
|
93
|
-
shutdown, supervision-window timeout), `pool-resume --id <pool> --from
|
|
94
|
-
<you>` re-attaches: it reconciles every launched worker (replied → done,
|
|
95
|
-
live → re-adopted, dead w/o reply → done with a `respawn` hint), then
|
|
96
|
-
supervises the unstarted remainder for a fresh window. Briefs are reused
|
|
97
|
-
by id, never duplicated; one supervisor at a time via an advisory pool
|
|
98
|
-
lock (TTL expiry lets a later attach take over after a crash).
|
|
99
|
-
`finishedAt` marks true completion only — a timed-out pool stays
|
|
100
|
-
resumable. Pools created before the resumable schema (no queue cursor)
|
|
101
|
-
are refused with guidance. Lead/admin only.
|
|
102
|
-
|
|
103
|
-
Windows note: console-less detached `cmd.exe` wrappers linger up to ~60s
|
|
104
|
-
after their payload exits, so natural-exit detection on `shell:true`
|
|
105
|
-
harnesses (opencode/claude/codex/cursor
|
|
106
|
-
minute; kills report immediately. Pool turnover and `spawn-status`
|
|
107
|
-
inherit the tail — size supervision windows accordingly.
|
|
108
|
-
|
|
109
|
-
## Long runs (the 10-day task)
|
|
110
|
-
|
|
111
|
-
Don't run one agent for 10 days — run a board for 10 days. Full playbook:
|
|
112
|
-
`docs/LONG_RUNS.md` (setup checklist, wave pattern, recovery ladder).
|
|
113
|
-
The short version: sessions are a cache, the board is the database —
|
|
114
|
-
rotate workers in waves, checkpoint progress, poll digest-first, persist
|
|
115
|
-
tokens to files, bound everything, and let compaction/restart/reboot
|
|
116
|
-
happen. A cold session plus a warm board is a crew that never sleeps.
|
|
117
|
-
|
|
118
|
-
## Unacknowledged-brief timeout (retry / reassign)
|
|
119
|
-
|
|
120
|
-
`ack` is orthogonal to delivery: a reply that sits `--unacked` is work
|
|
121
|
-
nobody accepted. The reassign pattern:
|
|
122
|
-
|
|
123
|
-
```sh
|
|
124
|
-
# what is still open — and how long has it waited?
|
|
125
|
-
crewbus inbox --from lead --unacked --older-than 10m
|
|
126
|
-
|
|
127
|
-
# the brief is a file: re-brief a fresh worker on the same thread
|
|
128
|
-
crewbus send --from lead --to w9 --reply <brief-id> --body "picking this up (w3 went silent)"
|
|
129
|
-
|
|
130
|
-
# the dead watcher's mail, if any, can be replayed
|
|
131
|
-
crewbus redeliver --from w3 --all
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
There is no automatic reaper: leads own the timeout policy for their
|
|
135
|
-
crew. `spawn --timeout 10m` records `deadlineAt` (`spawn-status` flags
|
|
136
|
-
passed deadlines — reap with `spawn-kill`/`stop`); `stop --all --from <you>`
|
|
137
|
-
terminates everything when the whole run is lost.
|
|
138
|
-
|
|
139
|
-
## Sync partitions
|
|
140
|
-
|
|
141
|
-
- Message files are immutable with unique ids: sync is **conflict-free
|
|
142
|
-
union** — both sides end up with both sides' mail.
|
|
143
|
-
- Presence, cursors and groups merge by **hybrid logical clock +
|
|
144
|
-
version counter** (wall-mtime fallback for legacy docs); newer
|
|
145
|
-
`(hlc, v)` wins, ties keep local content and converge clocks so the
|
|
146
|
-
next round is `pulled 0, pushed 0`.
|
|
147
|
-
- **Prune stays local.** Deletions replicate only as **tombstones**
|
|
148
|
-
(`tombstones/*.json`, synced): a pruned message never resurrects on a
|
|
149
|
-
partitioned peer, but the peer's other history is untouched.
|
|
150
|
-
- Rounds after the first are incremental (per-peer cursor, 60s overlap);
|
|
151
|
-
`--dry-run` previews pulls/pushes/tombstones without touching files.
|
|
152
|
-
|
|
153
|
-
## Worker lifetime
|
|
154
|
-
|
|
155
|
-
- `spawn` defaults to **oneshot**: headless workers (`claude -p`,
|
|
156
|
-
`codex exec`, `cursor-agent -p`, …) run the brief and exit. Their mail
|
|
157
|
-
**waits for pull** — a finished worker cannot receive push.
|
|
158
|
-
- `--persistent` marks long-lived peers (opencode sessions, daemons)
|
|
159
|
-
that stay addressable. `spawn-status` shows the recorded lifetime.
|
|
160
|
-
- Either way: **finished workers' DMs are files** — `inbox --from <name>`
|
|
161
|
-
reads them after the process is gone.
|
|
162
|
-
- **Session-id capture (respawn prerequisite).** Spawn boots the four
|
|
163
|
-
JSON-output harnesses with machine-readable logs (`opencode run
|
|
164
|
-
--format json`, `claude -p --output-format json`, `grok
|
|
165
|
-
--output-format json`, `codex exec --json`) so the harness session id
|
|
166
|
-
can be recovered from the worker log. `spawn-status` lazily extracts it
|
|
167
|
-
into `worker-sessions/<name>.json` (kept beside — not inside — the
|
|
168
|
-
agent doc, so presence heartbeats can never wipe it; machine-local,
|
|
169
|
-
never synced). Provenance is tracked (`preassigned` /
|
|
170
|
-
`preassigned-confirmed` / `log` / `log-override` — a disagreeing log
|
|
171
|
-
wins, in case the installed harness ignored the flag).
|
|
172
|
-
- **Pre-assigned ids (the mid-run-kill case).** Log parsing can't cover a
|
|
173
|
-
worker killed before emitting anything (notably grok, whose id only
|
|
174
|
-
arrives end-of-run). Where the CLI supports creating with an id,
|
|
175
|
-
`bootWorker` mints a UUID and passes it in (`claude --session-id`,
|
|
176
|
-
grok `-s`), so the binding holds a resumable id from boot. opencode
|
|
177
|
-
has no pre-set flag (its id arrives in the first stream event — the
|
|
178
|
-
residual window is milliseconds); codex relies on `thread.started`;
|
|
179
|
-
cursor/agy offer neither.
|
|
180
|
-
- **Respawn (`respawn --from <lead> --to <worker>`).** Reboots one dead
|
|
181
|
-
worker in its *same* harness conversation: `opencode run --session`,
|
|
182
|
-
`claude -p --resume` (+ `CLAUDE_CODE_RESUME_INTERRUPTED_TURN=1` so a
|
|
183
|
-
killed turn continues), `codex exec resume`, `grok -r`, agy
|
|
184
|
-
`--conversation`, cursor `--resume`;
|
|
185
|
-
|
|
186
|
-
and threads the same reply id, so the worker continues instead of
|
|
187
|
-
restarting; codex points at the catch-up file itself (its positional
|
|
188
|
-
prompt can't attach files). Refuses live workers unless `--force`
|
|
189
|
-
(kills first, own-crew scoped like `spawn-kill`); refuses unknown,
|
|
190
|
-
never-captured, and pre-capture-era workers with re-brief guidance.
|
|
191
|
-
Attempts count on the binding; `--dry-run` previews. Lead/admin only.
|
|
192
|
-
|
|
193
|
-
## Poll discipline (long runs)
|
|
194
|
-
|
|
195
|
-
Plain `inbox` dumps everything — over days that burns context.
|
|
196
|
-
Poll digest-first instead, escalate only on hits:
|
|
197
|
-
|
|
198
|
-
1. `crewbus inbox --from <you> --unacked --digest` — cheap
|
|
199
|
-
one-line-per-message sweep (`dm_inbox({ agent, unacked: true,
|
|
200
|
-
digest: true })` on MCP).
|
|
201
|
-
2. Filtered full read — same sweep narrowed before expanding:
|
|
202
|
-
`inbox --from <you> --unacked --grep <pat>` /
|
|
203
|
-
`--priority high` (MCP: `grep` / `priority` params).
|
|
204
|
-
3. `thread --id <msg-id>` / `gather` for context — only once a hit
|
|
205
|
-
proves worth reading.
|
|
206
|
-
|
|
207
|
-
Cadence: poll cheap and often; full reads on signal. Push wakes you;
|
|
208
|
-
digest tells you if it matters. Checkpoints compose: digests mark
|
|
209
|
-
progress notes `[checkpoint]` (skipped by `--unacked` triage, never
|
|
210
|
-
need ack), so the cheap poll still shows where a thread stands.
|
|
1
|
+
# Delivery semantics
|
|
2
|
+
|
|
3
|
+
How the bus behaves when things go wrong. Read this before building
|
|
4
|
+
orchestrators on top of `send`/`spawn`.
|
|
5
|
+
|
|
6
|
+
## At-least-once, not at-most-once
|
|
7
|
+
|
|
8
|
+
- **Files are the truth.** `send` writes one immutable file per DM
|
|
9
|
+
(`dm/<recipient>/<id>.json`) or one `broadcast/<batch>.json` for
|
|
10
|
+
fan-outs over 20. A send that echoes an id **landed**.
|
|
11
|
+
- **Push is at-least-once.** The opencode watcher and the hook helpers
|
|
12
|
+
claim `delivered/<agent>/<msg>.json` markers with an exclusive create,
|
|
13
|
+
and the cursor advances only on successful injection. A failed push
|
|
14
|
+
(e.g. into a stale session) releases the marker and is **retried**, so a
|
|
15
|
+
recipient may see a DM twice across a crash — never zero times.
|
|
16
|
+
- **Pull (`inbox`) is idempotent.** No mark-read side effects; page with
|
|
17
|
+
`--after`, filter with `--unacked`. `redeliver --id|--all` rewinds the
|
|
18
|
+
cursor after a dead watcher consumed mail.
|
|
19
|
+
|
|
20
|
+
## Ordering
|
|
21
|
+
|
|
22
|
+
- **No global order.** DMs from different senders interleave arbitrarily.
|
|
23
|
+
- **Per-sender order is wall-clock + id.** `inbox`/`gather`/`thread`
|
|
24
|
+
present oldest first by `at` timestamp; ids sort ties (4 random bytes,
|
|
25
|
+
unique under parallel sends).
|
|
26
|
+
- **Channels append.** `channels/<name>.log.jsonl` is append-only; concurrent
|
|
27
|
+
posts serialize on file append. Sync merges channel logs **union-by-id**,
|
|
28
|
+
so both sides' lines survive a partition.
|
|
29
|
+
|
|
30
|
+
## What happens when a worker dies mid-brief
|
|
31
|
+
|
|
32
|
+
- The **brief survives** — it is a file, not a process. A replacement
|
|
33
|
+
worker with the same name sees it via `inbox` (stale names are reusable;
|
|
34
|
+
spawning refuses names with a *live* worker behind them).
|
|
35
|
+
- A **detached worker's reply may never come.** `spawn-status` shows
|
|
36
|
+
`running? reply landed? acked?` plus a log tail: an exited worker with
|
|
37
|
+
no reply failed silently — read its log in `.crewbus/logs/`.
|
|
38
|
+
- **Pool mode** (`pool --count N --pool-size S`) replaces dead workers
|
|
39
|
+
automatically up to N total, with backpressure (queue refuses past
|
|
40
|
+
4× pool size).
|
|
41
|
+
|
|
42
|
+
## Presence you can trust (recycled-pid guard)
|
|
43
|
+
|
|
44
|
+
`spawn-status` liveness used to be `kill(pid, 0)` alone: after a reboot
|
|
45
|
+
the OS recycles pids and a dead worker could read as "running". Status
|
|
46
|
+
now also resolves the process start time (`ps` elapsed on unix,
|
|
47
|
+
`Get-Process` on Windows, 60s memo) and compares it against the recorded
|
|
48
|
+
`spawnedAt` (±120s skew): a live pid whose process started *after* the
|
|
49
|
+
spawn is a stranger, and `alive` flips to false. The verdict rides along
|
|
50
|
+
as `aliveVerified` (`true` corroborated, `false` contradicted, `null`
|
|
51
|
+
unverifiable on this platform — the old kill-0 verdict stands). The
|
|
52
|
+
override needs positive evidence (start time after spawn + skew), so a
|
|
53
|
+
false "dead" takes a >2min clock jump between spawn and check — rare,
|
|
54
|
+
and missing data never flips a verdict. Practical effect: after a
|
|
55
|
+
reboot, recycled pids read dead instead of running, so `respawn` is
|
|
56
|
+
allowed instead of wrongly refused. Reconcile sweeps (`doctor`,
|
|
57
|
+
post-reboot `spawn-status --all` review) remain the operator's job for
|
|
58
|
+
now.
|
|
59
|
+
|
|
60
|
+
## Progress checkpoints
|
|
61
|
+
|
|
62
|
+
Long briefs die mid-way; re-deriving progress from files wastes the
|
|
63
|
+
resume. Workers checkpoint every few steps (or before anything risky):
|
|
64
|
+
|
|
65
|
+
```sh
|
|
66
|
+
crewbus send --from <you> --to <lead> --reply <brief-id> --checkpoint --body "done X / next Y"
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
(`dm_send` takes `checkpoint: true`; same on the opencode `dm-send`
|
|
70
|
+
tool.) Checkpoints ride the normal DM thread, so `thread`/`gather`,
|
|
71
|
+
sync, and push delivery all work unchanged — but they are *progress*,
|
|
72
|
+
not work awaiting acceptance:
|
|
73
|
+
|
|
74
|
+
- `--unacked` (CLI and `dm_inbox`) and the `ack --timeout` hint skip
|
|
75
|
+
them; transcripts label them (`checkpoint: progress`, `[checkpoint]`
|
|
76
|
+
in digests).
|
|
77
|
+
- `spawn-status` reply detection ignores them: only a non-checkpoint
|
|
78
|
+
reply to the brief counts as "reply landed".
|
|
79
|
+
- `ack --all` still records them if run (harmless); rate limits and
|
|
80
|
+
duplicate suppression apply like any send, so checkpoint every few
|
|
81
|
+
steps — not every tool call.
|
|
82
|
+
|
|
83
|
+
A respawned worker reads the latest checkpoint off the thread
|
|
84
|
+
(`inbox`/`thread --id <brief-id>`) and continues from "next Y" instead
|
|
85
|
+
of restarting. Checkpoints are unverified progress: they never feed
|
|
86
|
+
`result record` / `race` winner logic (that still needs `ack --verify`).
|
|
87
|
+
|
|
88
|
+
## Pool supervision + re-attach
|
|
89
|
+
|
|
90
|
+
`pool` briefs all workers up front, boots at most `--pool-size`
|
|
91
|
+
concurrently, and replaces exits until N total — but the supervision loop
|
|
92
|
+
runs inside the invoking CLI process. If that process dies (lead restart,
|
|
93
|
+
shutdown, supervision-window timeout), `pool-resume --id <pool> --from
|
|
94
|
+
<you>` re-attaches: it reconciles every launched worker (replied → done,
|
|
95
|
+
live → re-adopted, dead w/o reply → done with a `respawn` hint), then
|
|
96
|
+
supervises the unstarted remainder for a fresh window. Briefs are reused
|
|
97
|
+
by id, never duplicated; one supervisor at a time via an advisory pool
|
|
98
|
+
lock (TTL expiry lets a later attach take over after a crash).
|
|
99
|
+
`finishedAt` marks true completion only — a timed-out pool stays
|
|
100
|
+
resumable. Pools created before the resumable schema (no queue cursor)
|
|
101
|
+
are refused with guidance. Lead/admin only.
|
|
102
|
+
|
|
103
|
+
Windows note: console-less detached `cmd.exe` wrappers linger up to ~60s
|
|
104
|
+
after their payload exits, so natural-exit detection on `shell:true`
|
|
105
|
+
harnesses (opencode/claude/codex/cursor) trails by up to a
|
|
106
|
+
minute; kills report immediately. Pool turnover and `spawn-status`
|
|
107
|
+
inherit the tail — size supervision windows accordingly.
|
|
108
|
+
|
|
109
|
+
## Long runs (the 10-day task)
|
|
110
|
+
|
|
111
|
+
Don't run one agent for 10 days — run a board for 10 days. Full playbook:
|
|
112
|
+
`docs/LONG_RUNS.md` (setup checklist, wave pattern, recovery ladder).
|
|
113
|
+
The short version: sessions are a cache, the board is the database —
|
|
114
|
+
rotate workers in waves, checkpoint progress, poll digest-first, persist
|
|
115
|
+
tokens to files, bound everything, and let compaction/restart/reboot
|
|
116
|
+
happen. A cold session plus a warm board is a crew that never sleeps.
|
|
117
|
+
|
|
118
|
+
## Unacknowledged-brief timeout (retry / reassign)
|
|
119
|
+
|
|
120
|
+
`ack` is orthogonal to delivery: a reply that sits `--unacked` is work
|
|
121
|
+
nobody accepted. The reassign pattern:
|
|
122
|
+
|
|
123
|
+
```sh
|
|
124
|
+
# what is still open — and how long has it waited?
|
|
125
|
+
crewbus inbox --from lead --unacked --older-than 10m
|
|
126
|
+
|
|
127
|
+
# the brief is a file: re-brief a fresh worker on the same thread
|
|
128
|
+
crewbus send --from lead --to w9 --reply <brief-id> --body "picking this up (w3 went silent)"
|
|
129
|
+
|
|
130
|
+
# the dead watcher's mail, if any, can be replayed
|
|
131
|
+
crewbus redeliver --from w3 --all
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
There is no automatic reaper: leads own the timeout policy for their
|
|
135
|
+
crew. `spawn --timeout 10m` records `deadlineAt` (`spawn-status` flags
|
|
136
|
+
passed deadlines — reap with `spawn-kill`/`stop`); `stop --all --from <you>`
|
|
137
|
+
terminates everything when the whole run is lost.
|
|
138
|
+
|
|
139
|
+
## Sync partitions
|
|
140
|
+
|
|
141
|
+
- Message files are immutable with unique ids: sync is **conflict-free
|
|
142
|
+
union** — both sides end up with both sides' mail.
|
|
143
|
+
- Presence, cursors and groups merge by **hybrid logical clock +
|
|
144
|
+
version counter** (wall-mtime fallback for legacy docs); newer
|
|
145
|
+
`(hlc, v)` wins, ties keep local content and converge clocks so the
|
|
146
|
+
next round is `pulled 0, pushed 0`.
|
|
147
|
+
- **Prune stays local.** Deletions replicate only as **tombstones**
|
|
148
|
+
(`tombstones/*.json`, synced): a pruned message never resurrects on a
|
|
149
|
+
partitioned peer, but the peer's other history is untouched.
|
|
150
|
+
- Rounds after the first are incremental (per-peer cursor, 60s overlap);
|
|
151
|
+
`--dry-run` previews pulls/pushes/tombstones without touching files.
|
|
152
|
+
|
|
153
|
+
## Worker lifetime
|
|
154
|
+
|
|
155
|
+
- `spawn` defaults to **oneshot**: headless workers (`claude -p`,
|
|
156
|
+
`codex exec`, `cursor-agent -p`, …) run the brief and exit. Their mail
|
|
157
|
+
**waits for pull** — a finished worker cannot receive push.
|
|
158
|
+
- `--persistent` marks long-lived peers (opencode sessions, daemons)
|
|
159
|
+
that stay addressable. `spawn-status` shows the recorded lifetime.
|
|
160
|
+
- Either way: **finished workers' DMs are files** — `inbox --from <name>`
|
|
161
|
+
reads them after the process is gone.
|
|
162
|
+
- **Session-id capture (respawn prerequisite).** Spawn boots the four
|
|
163
|
+
JSON-output harnesses with machine-readable logs (`opencode run
|
|
164
|
+
--format json`, `claude -p --output-format json`, `grok
|
|
165
|
+
--output-format json`, `codex exec --json`) so the harness session id
|
|
166
|
+
can be recovered from the worker log. `spawn-status` lazily extracts it
|
|
167
|
+
into `worker-sessions/<name>.json` (kept beside — not inside — the
|
|
168
|
+
agent doc, so presence heartbeats can never wipe it; machine-local,
|
|
169
|
+
never synced). Provenance is tracked (`preassigned` /
|
|
170
|
+
`preassigned-confirmed` / `log` / `log-override` — a disagreeing log
|
|
171
|
+
wins, in case the installed harness ignored the flag).
|
|
172
|
+
- **Pre-assigned ids (the mid-run-kill case).** Log parsing can't cover a
|
|
173
|
+
worker killed before emitting anything (notably grok, whose id only
|
|
174
|
+
arrives end-of-run). Where the CLI supports creating with an id,
|
|
175
|
+
`bootWorker` mints a UUID and passes it in (`claude --session-id`,
|
|
176
|
+
grok `-s`), so the binding holds a resumable id from boot. opencode
|
|
177
|
+
has no pre-set flag (its id arrives in the first stream event — the
|
|
178
|
+
residual window is milliseconds); codex relies on `thread.started`;
|
|
179
|
+
cursor/agy offer neither.
|
|
180
|
+
- **Respawn (`respawn --from <lead> --to <worker>`).** Reboots one dead
|
|
181
|
+
worker in its *same* harness conversation: `opencode run --session`,
|
|
182
|
+
`claude -p --resume` (+ `CLAUDE_CODE_RESUME_INTERRUPTED_TURN=1` so a
|
|
183
|
+
killed turn continues), `codex exec resume`, `grok -r`, agy
|
|
184
|
+
`--conversation`, cursor `--resume`; `--fresh` re-boots fresh without
|
|
185
|
+
prior conversation continuity. The catch-up brief points at the original prompt file
|
|
186
|
+
and threads the same reply id, so the worker continues instead of
|
|
187
|
+
restarting; codex points at the catch-up file itself (its positional
|
|
188
|
+
prompt can't attach files). Refuses live workers unless `--force`
|
|
189
|
+
(kills first, own-crew scoped like `spawn-kill`); refuses unknown,
|
|
190
|
+
never-captured, and pre-capture-era workers with re-brief guidance.
|
|
191
|
+
Attempts count on the binding; `--dry-run` previews. Lead/admin only.
|
|
192
|
+
|
|
193
|
+
## Poll discipline (long runs)
|
|
194
|
+
|
|
195
|
+
Plain `inbox` dumps everything — over days that burns context.
|
|
196
|
+
Poll digest-first instead, escalate only on hits:
|
|
197
|
+
|
|
198
|
+
1. `crewbus inbox --from <you> --unacked --digest` — cheap
|
|
199
|
+
one-line-per-message sweep (`dm_inbox({ agent, unacked: true,
|
|
200
|
+
digest: true })` on MCP).
|
|
201
|
+
2. Filtered full read — same sweep narrowed before expanding:
|
|
202
|
+
`inbox --from <you> --unacked --grep <pat>` /
|
|
203
|
+
`--priority high` (MCP: `grep` / `priority` params).
|
|
204
|
+
3. `thread --id <msg-id>` / `gather` for context — only once a hit
|
|
205
|
+
proves worth reading.
|
|
206
|
+
|
|
207
|
+
Cadence: poll cheap and often; full reads on signal. Push wakes you;
|
|
208
|
+
digest tells you if it matters. Checkpoints compose: digests mark
|
|
209
|
+
progress notes `[checkpoint]` (skipped by `--unacked` triage, never
|
|
210
|
+
need ack), so the cheap poll still shows where a thread stands.
|