bullswarm 0.10.9 → 0.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +37 -0
- package/docs/claude-dynamic-workflow-mechanics.md +231 -0
- package/docs/experiments/2026-08-29-ultracode-vs-bullswarm.md +240 -0
- package/package.json +1 -1
- package/skill/SKILL.md +8 -0
- package/src/help.js +1 -1
- package/src/workflow/cli.js +3 -1
- package/src/workflow/decision.js +40 -5
- package/src/workflow/goal.js +5 -5
- package/src/workflow/runner.js +59 -40
- package/src/workflow/runtime.js +20 -3
- package/src/workflow/watch-cli.js +18 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,42 @@
|
|
|
1
1
|
# bullswarm changelog
|
|
2
2
|
|
|
3
|
+
## 0.11.1 — reliable `--watch` handoff
|
|
4
|
+
|
|
5
|
+
- `workflow goal --watch` no longer races the detached child: the watcher now
|
|
6
|
+
waits up to 30 s for the run's `state.json` to appear before attaching, and
|
|
7
|
+
`runWorkflowWatch` accepts `waitForRunMs`. The 0.11.0 tag failed to publish
|
|
8
|
+
because this race made the release-gate test fail on the CI runner; 0.11.1
|
|
9
|
+
carries the full 0.11.0 change set below.
|
|
10
|
+
|
|
11
|
+
## 0.11.0 — plan the whole graph, run it wide
|
|
12
|
+
|
|
13
|
+
Adopts the driving mechanics of Claude Code's dynamic workflow (documented in
|
|
14
|
+
`docs/claude-dynamic-workflow-mechanics.md`) into the autonomous loop.
|
|
15
|
+
|
|
16
|
+
- Ready-set scheduler: every planner action whose dependencies have succeeded
|
|
17
|
+
starts immediately, and a dependent action starts the moment its own inputs
|
|
18
|
+
finish rather than when the whole round finishes. The global
|
|
19
|
+
`settings.concurrency` limiter caps real parallelism. Verify-B now overlaps
|
|
20
|
+
fix-C exactly like a Claude `pipeline()` stage.
|
|
21
|
+
- Planning doctrine: the planner is told to propose the complete dependency
|
|
22
|
+
graph in one decision (per-item fix→verify chains plus one whole-system
|
|
23
|
+
verify), to declare file ownership per action and order same-file edits with
|
|
24
|
+
`dependsOn`, to write self-contained worker prompts, and what a planning round
|
|
25
|
+
trip costs. The goal orchestrator prompt no longer asks for "the smallest
|
|
26
|
+
useful set" of actions. `executionConstraints.concurrency` is exposed.
|
|
27
|
+
- `workflow goal` default `--concurrency` is 8 (was 3; max 16).
|
|
28
|
+
- The planner prompt's shared-working-tree caution now says what is actually
|
|
29
|
+
unsafe (whole-tree mutation, running the full suite while others edit) and
|
|
30
|
+
states that concurrent workers editing disjoint files is the expected mode;
|
|
31
|
+
the 0.10.9 orchestrator had cited the old wording as its reason not to fan
|
|
32
|
+
out ("Implementation is deliberately NOT fanned out … shared-target mutation
|
|
33
|
+
policy").
|
|
34
|
+
- `verify.review` is recovered when a planner puts instructions or a filesystem
|
|
35
|
+
path there: instructions move to `prompt`, the single dependency's artifact is
|
|
36
|
+
inferred, and any `review` that is not `outputs.<actionId>.outFile` is
|
|
37
|
+
rejected at validation (feeding the corrective turn) instead of failing a
|
|
38
|
+
dispatch after a full planning round trip.
|
|
39
|
+
|
|
3
40
|
## 0.10.9 — planner self-correction and honest silence
|
|
4
41
|
|
|
5
42
|
- An invalid or non-JSON orchestrator decision no longer fails the run. The
|
|
@@ -0,0 +1,231 @@
|
|
|
1
|
+
# How Claude Code drives a dynamic workflow — mechanics, and what bullswarm adopts
|
|
2
|
+
|
|
3
|
+
Written 2026-08-29 from inside a Claude Code session that has the `Workflow`
|
|
4
|
+
tool ("ultracode") loaded, by the model that authors those workflows.
|
|
5
|
+
|
|
6
|
+
Every statement is tagged:
|
|
7
|
+
|
|
8
|
+
- **[SPEC]** — quoted or closely paraphrased from the `Workflow` tool contract
|
|
9
|
+
and the `workflow-authoring` reference as loaded into the session on
|
|
10
|
+
2026-08-29. This is the documented behaviour the orchestrating model is
|
|
11
|
+
told to rely on.
|
|
12
|
+
- **[OBSERVED]** — measured in this repository's experiments
|
|
13
|
+
(`docs/experiments/2026-08-29-ultracode-vs-bullswarm.md`).
|
|
14
|
+
- **[INFERRED]** — my reading of how the harness must behave to satisfy the
|
|
15
|
+
spec. Not confirmed by source; treat as a hypothesis.
|
|
16
|
+
|
|
17
|
+
## 0. The one-paragraph shape
|
|
18
|
+
|
|
19
|
+
Claude's orchestrator (the main-loop model) does **not** decide step-by-step at
|
|
20
|
+
runtime. It writes a *program* — a small JavaScript script — that declares the
|
|
21
|
+
phases and calls `agent(prompt, opts)` once per worker, wired together with
|
|
22
|
+
`pipeline()` / `parallel()` / plain loops. The harness executes that program
|
|
23
|
+
deterministically, spawning subagents concurrently up to a cap, and the model
|
|
24
|
+
reads the aggregate return value when the program finishes. Planning is
|
|
25
|
+
front-loaded into one authoring act; parallelism is explicit in the code; every
|
|
26
|
+
worker receives an individually written, self-contained prompt and (usually) a
|
|
27
|
+
JSON schema its answer must satisfy. Re-planning happens either as ordinary
|
|
28
|
+
code (loops, conditionals) inside the script, or between scripts when the model
|
|
29
|
+
reads a result and authors the next one. **[SPEC]**
|
|
30
|
+
|
|
31
|
+
bullswarm's `workflow goal`, by contrast, runs an LLM *at every checkpoint*: a
|
|
32
|
+
`decide` step proposes JSON actions, the runtime validates and executes them,
|
|
33
|
+
then asks the LLM again. **[SPEC — bullswarm source]** The rest of this document
|
|
34
|
+
is about which of Claude's mechanics that loop can adopt without giving up its
|
|
35
|
+
one advantage — the user supplies a goal, never a graph.
|
|
36
|
+
|
|
37
|
+
## 1. Mechanics, one at a time
|
|
38
|
+
|
|
39
|
+
### 1.1 The control plane is code, authored once **[SPEC]**
|
|
40
|
+
|
|
41
|
+
- A script must begin with a pure-literal `export const meta = { name,
|
|
42
|
+
description, phases: [{ title, detail }] }`; the body uses `phase()`,
|
|
43
|
+
`agent()`, `pipeline()`, `parallel()`, `log()`, `args`, `budget`, and
|
|
44
|
+
`workflow()` (one level of nesting).
|
|
45
|
+
- The tool is explicitly for "multi-step orchestration where control flow
|
|
46
|
+
should be deterministic (loops, conditionals, fan-out) rather than
|
|
47
|
+
model-driven."
|
|
48
|
+
- The recommended pattern is **hybrid**: "scout inline first (list the files,
|
|
49
|
+
find the channels, scope the diff) to discover the work-list, then call
|
|
50
|
+
Workflow to pipeline over it. You don't need to know the shape before the
|
|
51
|
+
*task* — only before the *orchestration step*."
|
|
52
|
+
- Multi-phase work is several workflows in sequence: "run several in sequence —
|
|
53
|
+
read each result before deciding the next phase. You stay in the loop; each
|
|
54
|
+
workflow is one well-scoped fan-out."
|
|
55
|
+
|
|
56
|
+
Consequence: between two agents inside one workflow there is **no model
|
|
57
|
+
round-trip**. The next agent starts the instant its inputs exist. **[INFERRED
|
|
58
|
+
from spec; consistent with OBSERVED timings]**
|
|
59
|
+
|
|
60
|
+
### 1.2 Phases are progress groups, not barriers **[SPEC]**
|
|
61
|
+
|
|
62
|
+
- `phase(title)` starts a new phase; "subsequent agent() calls are grouped
|
|
63
|
+
under this title in the progress display". `opts.phase` on `agent()` assigns
|
|
64
|
+
the group explicitly and exists precisely "to avoid races on the global
|
|
65
|
+
phase() state" inside `pipeline()`/`parallel()` stages.
|
|
66
|
+
- Titles are matched exactly against `meta.phases`; an unmatched title "just
|
|
67
|
+
gets its own progress group".
|
|
68
|
+
- Nothing about a phase synchronises work. The **only** barrier is
|
|
69
|
+
`parallel()`.
|
|
70
|
+
|
|
71
|
+
### 1.3 Parallelism primitives **[SPEC]**
|
|
72
|
+
|
|
73
|
+
- `pipeline(items, stage1, stage2, …)`: "run each item through all stages
|
|
74
|
+
independently, NO barrier between stages. Item A can be in stage 3 while item
|
|
75
|
+
B is still in stage 1. This is the DEFAULT for multi-stage work. Wall-clock =
|
|
76
|
+
slowest single-item chain, not sum-of-slowest-per-stage." A stage that throws
|
|
77
|
+
drops that item to `null` and skips its remaining stages.
|
|
78
|
+
- `parallel(thunks)`: "run tasks concurrently. This is a BARRIER … Use ONLY
|
|
79
|
+
when you genuinely need all results together." A barrier is justified only
|
|
80
|
+
when stage N needs cross-item context from all of stage N−1 (dedup/merge,
|
|
81
|
+
early exit on zero count, "compare with the other findings").
|
|
82
|
+
- Anti-barrier guidance is explicit: "'I need to flatten/map/filter first' —
|
|
83
|
+
do it inside a pipeline stage"; "'The stages are conceptually separate' —
|
|
84
|
+
that's what pipeline() models. Separate stages ≠ synchronized stages."
|
|
85
|
+
- Caps: "Concurrent agent() calls are capped at min(16, available CPUs − 2) per
|
|
86
|
+
workflow — excess calls queue and run as slots free up." Lifetime cap 1000
|
|
87
|
+
agents; ≤ 4096 items per `parallel()`/`pipeline()` call.
|
|
88
|
+
|
|
89
|
+
### 1.4 One prompt per agent, structured return **[SPEC]**
|
|
90
|
+
|
|
91
|
+
- `agent(prompt, opts)`; "Subagents are told their final text IS the return
|
|
92
|
+
value (not a human-facing message), so they return raw data."
|
|
93
|
+
- `opts.schema` (JSON Schema): "the subagent is forced to call a
|
|
94
|
+
StructuredOutput tool and agent() returns the validated object — no parsing
|
|
95
|
+
needed … validation happens at the tool-call layer so the model retries on
|
|
96
|
+
mismatch."
|
|
97
|
+
- Other per-agent knobs: `label` (display), `phase` (group), `model`
|
|
98
|
+
(override; "default to omitting it"), `effort` (`low`…`max`), `isolation:
|
|
99
|
+
'worktree'` ("EXPENSIVE … use ONLY when agents mutate files in parallel and
|
|
100
|
+
would otherwise conflict"), `agentType` (custom subagent definition).
|
|
101
|
+
- `agent()` "returns null if the user skips the agent mid-run or the subagent
|
|
102
|
+
dies on a terminal API error after retries (filter with .filter(Boolean))."
|
|
103
|
+
|
|
104
|
+
### 1.5 Failure semantics live in code **[SPEC]**
|
|
105
|
+
|
|
106
|
+
- `parallel()` "never rejects" — a failing thunk becomes `null`.
|
|
107
|
+
- Retry, repair and convergence are ordinary loops: *loop-until-count*,
|
|
108
|
+
*loop-until-budget*, *loop-until-dry* ("keep spawning finders until K
|
|
109
|
+
consecutive rounds return nothing new"). No planner turn is spent deciding to
|
|
110
|
+
retry; the script already says so.
|
|
111
|
+
|
|
112
|
+
### 1.6 Determinism and resume **[SPEC]**
|
|
113
|
+
|
|
114
|
+
- `Date.now()`, `Math.random()`, argless `new Date()` throw in scripts ("they
|
|
115
|
+
would break resume"); timestamps come in via `args`.
|
|
116
|
+
- Every invocation persists its script; the run's `journal.jsonl` "records each
|
|
117
|
+
agent's actual return value". Resume = same script + `resumeFromRunId`: "the
|
|
118
|
+
longest unchanged prefix of agent() calls returns cached results instantly;
|
|
119
|
+
the first edited/new call and everything after it runs live."
|
|
120
|
+
|
|
121
|
+
### 1.7 Budget is a hard ceiling **[SPEC]**
|
|
122
|
+
|
|
123
|
+
- `budget.total / spent() / remaining()`; "The target is a HARD ceiling, not
|
|
124
|
+
advisory: once spent() reaches total, further agent() calls throw."
|
|
125
|
+
- bullswarm deliberately does the opposite (advisory targets, by user
|
|
126
|
+
decision: "a failed run is more costly than a slightly over budget run").
|
|
127
|
+
This difference is kept on purpose.
|
|
128
|
+
|
|
129
|
+
### 1.8 Observability **[SPEC]**
|
|
130
|
+
|
|
131
|
+
- Progress is rendered as a tree grouped by phase, with per-agent labels;
|
|
132
|
+
`log()` "emit[s] a progress message to the user (shown as a narrator line
|
|
133
|
+
above the progress tree)"; `/workflows` shows live progress; the tool result
|
|
134
|
+
carries the `runId` and transcript directory.
|
|
135
|
+
|
|
136
|
+
### 1.9 Quality patterns the orchestrator is told to compose **[SPEC]**
|
|
137
|
+
|
|
138
|
+
Adversarial verify (N skeptics prompted to refute; kill on majority),
|
|
139
|
+
perspective-diverse verify (distinct lenses instead of N identical refuters),
|
|
140
|
+
judge panel (N independent attempts → parallel judges → synthesis), loop-until-
|
|
141
|
+
dry, multi-modal sweep, completeness critic, and "no silent caps: if a workflow
|
|
142
|
+
bounds coverage (top-N, no-retry, sampling), log() what was dropped."
|
|
143
|
+
|
|
144
|
+
### 1.10 Ultracode **[SPEC]**
|
|
145
|
+
|
|
146
|
+
"When a system-reminder confirms ultracode is on, that opt-in is standing:
|
|
147
|
+
author and run a workflow for every substantive task by default … For
|
|
148
|
+
multi-phase work (understand → design → implement → review), that often means
|
|
149
|
+
several workflows in sequence — one per phase — so you stay in the loop between
|
|
150
|
+
them."
|
|
151
|
+
|
|
152
|
+
## 2. What that looks like from the outside **[OBSERVED]**
|
|
153
|
+
|
|
154
|
+
Filled from the experiment report as runs complete. Numbers here are copied
|
|
155
|
+
from `docs/experiments/2026-08-29-ultracode-vs-bullswarm.md`, never projected.
|
|
156
|
+
|
|
157
|
+
- bullswarm 0.10.9 smoke goal ("create hello.txt and verify it"), single pool
|
|
158
|
+
`claude-code` pinned to `claude-opus-5`: wall 553 s; **4 orchestrator turns =
|
|
159
|
+
438 s (79 % of wall)**; 2 worker attempts = 114 s; max concurrency 1; 6
|
|
160
|
+
dispatches; ~29 k tokens (bullswarm's byte/4 estimate). Two of the four turns
|
|
161
|
+
were spent recovering from a `verify` proposal whose `review` field carried
|
|
162
|
+
instructions instead of an artifact path — a shape the 0.10.9 planner
|
|
163
|
+
skeleton itself had suggested.
|
|
164
|
+
- bullswarm 0.10.9 on the 7-module fixture (baseline, `--concurrency 8`): the
|
|
165
|
+
orchestrator's first decision (152 s) proposed one serial chain
|
|
166
|
+
discover → implement → verify and wrote: "Implementation is deliberately NOT
|
|
167
|
+
fanned out: all fixes land in one shared working tree and converge on
|
|
168
|
+
src/index.js, so concurrent workers would violate the shared-target mutation
|
|
169
|
+
policy and race on the barrel file." The policy it cites was a caution line
|
|
170
|
+
in the planner prompt; a concurrency cap of 8 was available and unused.
|
|
171
|
+
Discovery alone then ran 458 s. (Final numbers: experiment report.)
|
|
172
|
+
- (comparison fixture final results: pending)
|
|
173
|
+
|
|
174
|
+
## 3. bullswarm today, mechanic by mechanic
|
|
175
|
+
|
|
176
|
+
| Mechanic | Claude `Workflow` | bullswarm ≤ 0.10.9 | Gap |
|
|
177
|
+
| --- | --- | --- | --- |
|
|
178
|
+
| Control plane | Code, authored once; no model call between agents | LLM `decide` turn at every checkpoint; each turn is a fresh `claude -p --resume` process reading the full durable context | Structural. Reachable target: **one planning turn per replan-worthy event** (initial DAG; then only on failure/completion), not per action |
|
|
179
|
+
| Phases | Labels for grouping; never synchronise | Forward-only kebab-case names per action; also just labels | None |
|
|
180
|
+
| Parallelism | `pipeline` default, `parallel` barrier; cap min(16, CPUs−2) | `executeActions` ran dependency-ready siblings **serially** (`runner.js:558`); only `fanout` items ran concurrently; goal default concurrency 3 | **Fixed in 0.11.0** — ready-set scheduler + default 8 |
|
|
181
|
+
| Planner bias | Script author is told to fan out and default to pipeline | Goal prompt said "return needs_more_work with the **smallest useful set** of bounded … actions" (`goal.js:18`) and planner prompt said "keep actions cohesive" | **Fixed in 0.11.0** — "propose the COMPLETE dependency graph", per-item fix→verify chains, file ownership, self-contained prompts |
|
|
182
|
+
| Per-agent prompt | Self-contained, plus JSON schema enforced at tool layer | Planner-authored prompt; free-text answer, content-verified by heuristics; `verify` returns JSON verdict | Partial. Schema-enforced worker output is a candidate, not adopted yet |
|
|
183
|
+
| Failure handling | Loops in code; `null` on agent death | Planner replans (costly); 0.10.9 added corrective turns for invalid decisions and 0.11.0 recovers mis-shaped `verify.review` before dispatch | Improved; retry-in-code per action still absent |
|
|
184
|
+
| Determinism / resume | Journal of return values; prefix cache | Durable `state.json` + `events.jsonl` + action ledger; resume skips durable outputs | Equivalent |
|
|
185
|
+
| Budget | Hard ceiling | Advisory targets (user decision) | Intentional difference |
|
|
186
|
+
| Observability | Progress tree, narrator, `/workflows` | `watch` heartbeat (semantic quiet + agent-output quiet since 0.10.9), `tui`, events | Comparable |
|
|
187
|
+
| Isolation | `isolation: 'worktree'` per agent | Shared `addDir`; planner-declared file ownership | Candidate |
|
|
188
|
+
|
|
189
|
+
## 4. Adopted into bullswarm 0.11.0
|
|
190
|
+
|
|
191
|
+
1. **Ready-set scheduler** (`src/workflow/runner.js`, `executeActions`).
|
|
192
|
+
Every action whose `dependsOn` have all succeeded is launched immediately;
|
|
193
|
+
a dependent action starts the moment *its own* dependencies finish, not when
|
|
194
|
+
the whole round finishes. The global dispatch limiter
|
|
195
|
+
(`settings.concurrency`) caps real concurrency. This gives `pipeline`
|
|
196
|
+
semantics to any DAG the planner proposes: verify-B overlaps fix-C.
|
|
197
|
+
Test: `dependency-ready sibling actions run concurrently and dependents
|
|
198
|
+
start as soon as their own inputs finish` (`tests/workflow-adaptive.test.js`).
|
|
199
|
+
2. **Planning doctrine in the planner prompt** (`src/workflow/runtime.js`) and
|
|
200
|
+
goal orchestrator prompt (`src/workflow/goal.js`): propose the complete graph
|
|
201
|
+
in one decision; independent actions run concurrently; per-item fix→verify
|
|
202
|
+
chains plus one final whole-system verify; explicit file ownership per
|
|
203
|
+
action and `dependsOn` for any same-file edits; self-contained worker
|
|
204
|
+
prompts with absolute paths and the exact acceptance command. The prompt
|
|
205
|
+
also states the cost of a planning turn so the model can weigh it. The
|
|
206
|
+
context exposes `executionConstraints.concurrency` and
|
|
207
|
+
`readySiblingsRunConcurrently: true`.
|
|
208
|
+
3. **Default `--concurrency` 8** for `workflow goal` (was 3; max 16).
|
|
209
|
+
4. **`verify.review` contract made survivable**: instructions placed in
|
|
210
|
+
`review` are moved to `prompt` and the single dependency's artifact is
|
|
211
|
+
inferred; a `review` that is not `outputs.<actionId>.outFile` is rejected at
|
|
212
|
+
validation (so the 0.10.9 corrective turn fixes it) instead of failing a
|
|
213
|
+
dispatch after a planning round trip.
|
|
214
|
+
|
|
215
|
+
## 5. Not adopted (yet), and why
|
|
216
|
+
|
|
217
|
+
- **Schema-enforced worker output.** bullswarm's content verification and the
|
|
218
|
+
JSON `verify` verdict cover the failure mode today; adding per-action
|
|
219
|
+
`outputSchema` is the next step if planners keep re-asking workers for
|
|
220
|
+
structure.
|
|
221
|
+
- **Per-action worktree isolation.** File ownership declared by the planner is
|
|
222
|
+
cheaper and matches Claude's own guidance ("EXPENSIVE … use ONLY when agents
|
|
223
|
+
mutate files in parallel and would otherwise conflict").
|
|
224
|
+
- **Hard budgets.** Deliberately advisory; see 1.7.
|
|
225
|
+
- **Retry policy in the graph.** Claude writes repair loops as code. A per-
|
|
226
|
+
action `retryPolicy` proposed by the planner would remove one planner turn per
|
|
227
|
+
transient failure; not yet built.
|
|
228
|
+
- **Identical control plane.** bullswarm's orchestrator remains an LLM per
|
|
229
|
+
checkpoint because the user supplies only a goal. The convergence target is
|
|
230
|
+
therefore "few planning turns, full width between them", measured as
|
|
231
|
+
`plannerSec / wallSec` and `maxConcurrentAttempts` in the experiment report.
|
|
@@ -0,0 +1,240 @@
|
|
|
1
|
+
# Claude ultracode dynamic workflow vs bullswarm `workflow goal` — side-by-side (2026-08-29)
|
|
2
|
+
|
|
3
|
+
Status: IN PROGRESS — results sections are filled in as runs complete. Numbers
|
|
4
|
+
below are observed, not projected; anything estimated is labelled.
|
|
5
|
+
|
|
6
|
+
## Question
|
|
7
|
+
|
|
8
|
+
Same goal, same models, same fixture: how does bullswarm's autonomous
|
|
9
|
+
`workflow goal` loop behave next to Claude Code's native dynamic workflow
|
|
10
|
+
(the `Workflow` tool, "ultracode"), and what has to change in bullswarm to make
|
|
11
|
+
it feel as smooth?
|
|
12
|
+
|
|
13
|
+
## Setup (identical on both sides)
|
|
14
|
+
|
|
15
|
+
- **Fixture**: `swarmbench`, a zero-dependency Node ESM project with six small
|
|
16
|
+
pure-logic modules under `src/`, one `node:test` file per module under
|
|
17
|
+
`tests/`, plus an intentionally incomplete `src/index.js` barrel with its own
|
|
18
|
+
test. Bugs were planted by a separate subagent; the orchestrator of the Claude
|
|
19
|
+
side (this session) was told only file names and baseline counts, never the
|
|
20
|
+
bugs, so both orchestrators had to discover them.
|
|
21
|
+
- **Two byte-identical copies**, each `git init`-ed so the diff can be audited
|
|
22
|
+
afterwards. Test files are SHA-256 fingerprinted before and after each run to
|
|
23
|
+
prove neither side edited a test.
|
|
24
|
+
- **Goal text (verbatim, both sides)**: see "Goal" below.
|
|
25
|
+
- **Models locked to one account**: workers on both sides are `claude-opus-5`.
|
|
26
|
+
Orchestrator: Claude side = this Claude Code session (Fable 5, authoring and
|
|
27
|
+
running the workflow script); bullswarm side = `claude-code` connector pinned
|
|
28
|
+
with `--orchestrator claude-code` and `strategy assign high|medium|low --pool
|
|
29
|
+
claude-code --model claude-opus-5`, `claude-fable-5` excluded.
|
|
30
|
+
- **bullswarm** 0.10.9 (released and installed from npm for this experiment),
|
|
31
|
+
run from the installed binary with an isolated `BULLSWARM_HOME` in which every
|
|
32
|
+
pool except `claude-code` is disabled and the connector meter is `none` so a
|
|
33
|
+
5-hour burst gate cannot bench the only pool mid-run.
|
|
34
|
+
- **Sequential runs** on the same laptop so wall-clock numbers are not skewed
|
|
35
|
+
by contention for the same subscription and CPUs.
|
|
36
|
+
|
|
37
|
+
## Goal
|
|
38
|
+
|
|
39
|
+
Verbatim, passed as the single goal argument to bullswarm and as the task
|
|
40
|
+
statement the Claude-side workflow script decomposes:
|
|
41
|
+
|
|
42
|
+
```text
|
|
43
|
+
Make `npm test` pass in this repository. Every failing test is caused by a bug in a file under src/ (including an incomplete src/index.js barrel that must re-export every module's public API). Fix the source so all tests pass. You must not modify, delete, or add any file under tests/. Finish by running the full `npm test` and report, per file changed, what was wrong and what you changed.
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## Fixture baseline
|
|
47
|
+
|
|
48
|
+
`swarmbench`: `src/{csv,duration,index,intervals,lru,semver,slugify}.js`,
|
|
49
|
+
`tests/{csv,duration,index,intervals,lru,semver,slugify}.test.js`, `npm test`
|
|
50
|
+
= `node --test tests/*.test.js`. Baseline on both copies (Node v22.23.2):
|
|
51
|
+
|
|
52
|
+
```text
|
|
53
|
+
# tests 52
|
|
54
|
+
# pass 37
|
|
55
|
+
# fail 15 (failures present in all 7 test files)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Test-file SHA-256 before any run (identical in both copies):
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
5106804683c7f6cef0dbc3629c4c14f4ab2e88319d934626d79bb18a080ae707 tests/csv.test.js
|
|
62
|
+
b7c7d77daad8bfe03dd0b755574cf59faaf677df7d2701a717cf8e44138bd108 tests/duration.test.js
|
|
63
|
+
27aa49f0f601d22748f36e161c963ac77bd9861fe84028a69e02a456a9b38302 tests/index.test.js
|
|
64
|
+
80abeecbea763df43b2260a7ee03df211db5b092ea87e418e0be740c59383cbd tests/intervals.test.js
|
|
65
|
+
2aa610981dcb9a6c65a0dd9b7b3362f23c95ef5c6436a43cb3d969fc0cc3dded tests/lru.test.js
|
|
66
|
+
d62bc016abe37f4400499f4a77b94c7b60cc8f473bac6b63f5a461f9a18b6a20 tests/semver.test.js
|
|
67
|
+
d611fd789caed8820eaf03b08aeb2799f564e5f0d320485f886fd9c5cf6ac112 tests/slugify.test.js
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Both bullswarm runs and the Claude workflow use a concurrency cap of 8
|
|
71
|
+
(Claude's own cap on this 10-CPU laptop is min(16, CPUs − 2) = 8; bullswarm
|
|
72
|
+
gets `--concurrency 8`), so parallelism differences come from the driving
|
|
73
|
+
loop, not the cap.
|
|
74
|
+
|
|
75
|
+
## Warm-up: bullswarm 0.10.9 smoke goal (observed)
|
|
76
|
+
|
|
77
|
+
Goal: "Create a file named hello.txt in this directory containing exactly the
|
|
78
|
+
single line 'hello from smoke test', then verify it exists with that exact
|
|
79
|
+
content." Run `srdwjs`, single pool `claude-code` pinned to `claude-opus-5`,
|
|
80
|
+
`--max-agents 5 --max-expansion-rounds 2 --concurrency 2`.
|
|
81
|
+
|
|
82
|
+
| metric | value |
|
|
83
|
+
| --- | --- |
|
|
84
|
+
| wall | 553 s (9m13s) |
|
|
85
|
+
| dispatches | 6 (4 orchestrator turns, 1 worker, 1 verifier) |
|
|
86
|
+
| orchestrator time | 438 s = 79 % of wall (turns: 108 s, 102 s, 151 s, 77 s) |
|
|
87
|
+
| worker time | 38 s (create) + 76 s (verify) |
|
|
88
|
+
| max concurrent attempts | 1 |
|
|
89
|
+
| tokens | ≈29.1 k (bullswarm estimate: utf8 bytes / 4) |
|
|
90
|
+
| outcome | `completed`, verified |
|
|
91
|
+
|
|
92
|
+
Two of the four planning turns were spent on `verify` actions that died inside
|
|
93
|
+
the runtime without dispatching: the planner put reviewer instructions (turn 2)
|
|
94
|
+
and then a filesystem path (turn 3) into `verify.review`, which the runtime
|
|
95
|
+
resolves as a dotted scope path to an artifact. The 0.10.9 planner skeleton had
|
|
96
|
+
shown exactly that wrong shape. Fixed in the 0.11.0 work (see below).
|
|
97
|
+
|
|
98
|
+
## Results
|
|
99
|
+
|
|
100
|
+
### bullswarm 0.10.9 `workflow goal` (baseline, installed binary)
|
|
101
|
+
|
|
102
|
+
Run `wf-mtd6lxfn-3912fd`, started 2026-08-28T16:44:18Z, `--orchestrator
|
|
103
|
+
claude-code --concurrency 8 --max-agents 30 --max-expansion-rounds 8`.
|
|
104
|
+
|
|
105
|
+
First planning turn: 152 s. It proposed one strictly serial chain —
|
|
106
|
+
`discover-failures` → `implement-fixes` → `verify-suite` — and explained why
|
|
107
|
+
(verbatim from `state.json` `decisions[0].reason`):
|
|
108
|
+
|
|
109
|
+
> Implementation is deliberately NOT fanned out: all fixes land in one shared
|
|
110
|
+
> working tree and converge on src/index.js, so concurrent workers would violate
|
|
111
|
+
> the shared-target mutation policy and race on the barrel file.
|
|
112
|
+
|
|
113
|
+
The "shared-target mutation policy" it cites is a caution line in the 0.10.9
|
|
114
|
+
planner prompt. `discover-failures` (read-only diagnosis of 7 test files) then
|
|
115
|
+
ran alone for 458 s before the single `implement-fixes` worker started.
|
|
116
|
+
|
|
117
|
+
Final numbers (run `xnujua`, 16:44:18Z → 17:19:26Z):
|
|
118
|
+
|
|
119
|
+
| metric | value |
|
|
120
|
+
| --- | --- |
|
|
121
|
+
| outcome | `completed`, verified; **52/52 tests pass**; `tests/` byte-identical (SHA-256 unchanged); 7 `src/` files changed, +20/−14 lines |
|
|
122
|
+
| wall | **2108 s (35m08s)** |
|
|
123
|
+
| dispatches | 7 = 3 orchestrator turns + 4 workers |
|
|
124
|
+
| orchestrator time | 431 s (20 %) — turns of 152 s, 189 s, 90 s |
|
|
125
|
+
| worker time | 1677 s: discover 458 s → implement 356 s → (verify-suite: runtime failure, 0 s) → independent-audit 618 s → final-verification 245 s |
|
|
126
|
+
| max concurrent attempts | **1** (parallelism 1.00 — every second of the run had exactly one process working) |
|
|
127
|
+
| planning rounds | 2 expansion rounds + 1 completion |
|
|
128
|
+
| tokens | ≈35.8 k by bullswarm's utf8-bytes/4 estimate of outputs only — **not comparable** to real provider usage (the Claude session's own status line showed ~49 k input tokens after its first four tool calls) |
|
|
129
|
+
|
|
130
|
+
Timeline of attempts (UTC):
|
|
131
|
+
|
|
132
|
+
```text
|
|
133
|
+
16:44:18 → 16:46:50 orchestrator turn 1 (152 s) → serial chain of 3
|
|
134
|
+
16:46:50 → 16:54:28 discover-failures (458 s) read-only diagnosis
|
|
135
|
+
16:54:28 → 17:00:24 implement-fixes (356 s) one worker fixes all 7 files; 52/52
|
|
136
|
+
verify-suite failed inside runtime (review = instructions), never dispatched
|
|
137
|
+
17:00:24 → 17:03:33 orchestrator turn 2 (189 s) → independent-audit + final-verification
|
|
138
|
+
17:03:33 → 17:13:51 independent-audit (618 s) re-tests, fuzzes (3 000 interval sets, 2 000 LRU trials, …)
|
|
139
|
+
17:13:51 → 17:17:57 final-verification (245 s) verify verdict ok
|
|
140
|
+
17:17:57 → 17:19:26 orchestrator turn 3 (90 s) complete
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
What the orchestrator did well: it delivered a correct result with an honest,
|
|
144
|
+
detailed completion report, refused to fan out for a stated reason, and
|
|
145
|
+
recovered from its own `verify.review` mistake ("it was my control-plane
|
|
146
|
+
error") in one extra turn. What cost time: a strictly serial plan under a
|
|
147
|
+
concurrency cap of 8 that was never used; a 458 s read-only discovery step
|
|
148
|
+
before any fix; a 618 s audit that fuzzed every module; and one wasted
|
|
149
|
+
planning round on the `verify.review` shape.
|
|
150
|
+
|
|
151
|
+
### Claude Code, fresh interactive session (Opus, ultracode)
|
|
152
|
+
|
|
153
|
+
Set-up: a brand-new Claude Code 2.1.250 session started by `herdr` in its own
|
|
154
|
+
tab (`claude-compare`), `--model claude-opus-5 --dangerously-skip-permissions`
|
|
155
|
+
(the same permission posture bullswarm's connector uses), `/effort ultracode`
|
|
156
|
+
confirmed on screen, cwd = the twin fixture copy, no prior context. The goal
|
|
157
|
+
text above was submitted as the first and only prompt at 17:19:16 Z. The
|
|
158
|
+
bullswarm baseline's final single-process planning turn was still running
|
|
159
|
+
until 17:19:26 Z — a 10-second overlap, noted for honesty.
|
|
160
|
+
|
|
161
|
+
Result (session `309931b0…`, goal turn 17:19:16 Z → 17:22:45 Z):
|
|
162
|
+
|
|
163
|
+
| metric | value |
|
|
164
|
+
| --- | --- |
|
|
165
|
+
| outcome | **52/52 tests pass**; `tests/` byte-identical (SHA-256 unchanged); 7 `src/` files changed, +21/−14 lines; per-file bug explanation printed |
|
|
166
|
+
| wall | **209 s (3m29s)** — the session's own footer: "Baked for 3m 29s" |
|
|
167
|
+
| agents spawned | **0** — no `Workflow`, no `Agent` call; 11 `Bash` calls in total |
|
|
168
|
+
| planning round trips | 0 |
|
|
169
|
+
| how it decided | 17:19:19–17:20:34 read every `src/` and `tests/` file and ran `npm test` inline; 17:20:34 "I have a complete picture: 15 failures across 7 modules. Let me get a review before I start editing." → advisor: apply directly, "the ultracode carve-out for trivial mechanical edits applies now that diagnosis is done"; 17:22:09 one Python patch script over all 7 files; 17:22:14–17:22:29 `npm test`, `git diff -- tests/` (empty), `git diff -- src/` |
|
|
170
|
+
| tokens (real provider usage from the transcript) | output 41 042; cache-read 1 939 219; cache-write 119 052; uncached input 62 — 24 assistant messages |
|
|
171
|
+
|
|
172
|
+
**The finding that matters for this experiment:** a fresh Opus session with
|
|
173
|
+
ultracode on judged this goal *too small to orchestrate* and did it solo. The
|
|
174
|
+
"smoothness" here is a fast inline scout (75 s to a complete diagnosis) and
|
|
175
|
+
zero planning round trips — not parallel agents. bullswarm's orchestrator, by
|
|
176
|
+
contrast, cannot do anything inline: every observation costs a worker
|
|
177
|
+
dispatch and every decision a full planning turn, so the same 15 bugs took
|
|
178
|
+
7 dispatches and 35 minutes.
|
|
179
|
+
|
|
180
|
+
Goal #1 therefore cannot show the fan-out mechanic the user asked about. Goal
|
|
181
|
+
#2 below is sized to require it.
|
|
182
|
+
|
|
183
|
+
### Correctness audit — goal #1
|
|
184
|
+
|
|
185
|
+
Independent `npm test` after each run: bullswarm copy 52/52, Claude copy
|
|
186
|
+
52/52. All seven `tests/*.test.js` SHA-256 values identical to the pre-run
|
|
187
|
+
fingerprints on both copies. Diff stat: bullswarm +20/−14 across the same 7
|
|
188
|
+
`src/` files; Claude +21/−14. Both fixed the same bugs the same way.
|
|
189
|
+
|
|
190
|
+
## Goal #2 — sized to require fan-out
|
|
191
|
+
|
|
192
|
+
Base: the fixed fixture (52/52) committed clean, copied three times
|
|
193
|
+
(`g2-claude`, `g2-bs-v2`, `g2-bs-base`). Goal text (verbatim, both sides):
|
|
194
|
+
|
|
195
|
+
```text
|
|
196
|
+
This repository (swarmbench) has six modules under src/ — csv, duration, intervals, lru, semver, slugify — and a passing test suite. For EACH of the six modules deliver three things: (1) a new test file tests/<module>.edge.test.js using node:test and node:assert/strict with at least 6 additional edge-case tests that are not already covered by tests/<module>.test.js, all passing against the current implementation; (2) complete JSDoc on every exported function or class in src/<module>.js (parameter types, return type, thrown errors, one @example) without changing any behaviour; (3) a documentation page docs/<module>.md describing the public API, edge-case behaviour, and examples. Then write docs/README.md as an index table linking all six pages with a one-line summary each. Do not modify the existing tests/<module>.test.js files. Finish with the full `npm test` passing (existing plus new tests) and report what you created.
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
Six independent items, three deliverables each, one cross-cutting index — the
|
|
200
|
+
shape `pipeline()` exists for.
|
|
201
|
+
|
|
202
|
+
### Claude Code, fresh interactive session #2 (Opus, ultracode)
|
|
203
|
+
|
|
204
|
+
New session (`0754e69f…`) in herdr tab `claude-compare-2`, same set-up as
|
|
205
|
+
session #1; goal submitted 17:24:4x Z. Observed driving sequence:
|
|
206
|
+
|
|
207
|
+
- 17:24:50–17:25:20 inline scouting (ls, `cat -n` every source and test file,
|
|
208
|
+
`npm test` → 52 passing). First words: "I'll start by orienting myself in the
|
|
209
|
+
repo before authoring the workflow."
|
|
210
|
+
- 17:25:20–17:26:44 advisor consultation ("Advisor sharpened the plan"); then
|
|
211
|
+
`mkdir docs` and a baseline snapshot of `src/` for behaviour-preservation
|
|
212
|
+
checks.
|
|
213
|
+
- 17:28:42 `Workflow` call: script `swarmbench-docs-and-edge-tests`, phases
|
|
214
|
+
Probe → Author → Verify, five `agent()` call sites in a per-module
|
|
215
|
+
`pipeline()` ("probe → author → adversarially verify → fix-loop, pipelined
|
|
216
|
+
per module"), ≈23 k characters, workers inherit the session model (Opus).
|
|
217
|
+
- The first call was **rejected by the harness**: "Invalid workflow script:
|
|
218
|
+
Script parse error: Unexpected token (48:15) … Workflow scripts must be plain
|
|
219
|
+
JavaScript". The session replied "Backtick inside a template literal in the
|
|
220
|
+
lru hints. Fixing and resending." — the same validate-and-correct loop
|
|
221
|
+
bullswarm 0.10.9 now has for planner decisions, except here the correction is
|
|
222
|
+
an inline retry inside one turn rather than a fresh planning process.
|
|
223
|
+
|
|
224
|
+
(execution numbers pending)
|
|
225
|
+
|
|
226
|
+
### bullswarm 0.11.0 code (`node bin/bullswarm.js` from the repo checkout)
|
|
227
|
+
|
|
228
|
+
(pending)
|
|
229
|
+
|
|
230
|
+
### bullswarm 0.10.9 (installed binary), if time allows
|
|
231
|
+
|
|
232
|
+
(pending)
|
|
233
|
+
|
|
234
|
+
## Behaviour differences observed
|
|
235
|
+
|
|
236
|
+
(pending)
|
|
237
|
+
|
|
238
|
+
## What to change in bullswarm
|
|
239
|
+
|
|
240
|
+
(pending)
|
package/package.json
CHANGED
package/skill/SKILL.md
CHANGED
|
@@ -332,6 +332,14 @@ optional work. The loop is durable:
|
|
|
332
332
|
execute -> observe -> decide -> validate proposal -> append -> execute -> observe
|
|
333
333
|
```
|
|
334
334
|
|
|
335
|
+
Execution of an accepted proposal is a ready-set scheduler: every action whose
|
|
336
|
+
`dependsOn` have succeeded starts at once (up to `settings.concurrency`, default
|
|
337
|
+
8 for `workflow goal`), and a dependent action starts the moment its own inputs
|
|
338
|
+
finish. The planner is instructed to propose the complete graph per decision —
|
|
339
|
+
per-item fix→verify chains plus one whole-system verify — because each planning
|
|
340
|
+
turn is a full orchestrator round trip. See
|
|
341
|
+
`docs/claude-dynamic-workflow-mechanics.md` for the model this follows.
|
|
342
|
+
|
|
335
343
|
Allowed planner decisions are `proceed`, `complete`, `needs_more_work`,
|
|
336
344
|
`retry`, `escalate`, `wait_for_approval`, and `stop`. Expansion decisions must
|
|
337
345
|
contain bounded actions; malformed or over-budget output executes nothing.
|
package/src/help.js
CHANGED
|
@@ -587,7 +587,7 @@ const workflowGoalText = rich({
|
|
|
587
587
|
{ flag: '--max-actions <n>', desc: 'planning target for total dispatched actions', default: '40 (max 1000)' },
|
|
588
588
|
{ flag: '--max-items-per-expansion <n>', desc: 'cap on fanout items added per planner round', default: '8 (max 100)' },
|
|
589
589
|
{ flag: '--max-workflow-seconds <n>', desc: 'planning target for total wall-clock seconds', default: '3600 (max 86400)' },
|
|
590
|
-
{ flag: '--concurrency <n>', desc: 'max parallel dispatches', default: '
|
|
590
|
+
{ flag: '--concurrency <n>', desc: 'max parallel dispatches; dependency-ready actions from one decision run concurrently up to this cap', default: '8 (max 16)' },
|
|
591
591
|
{ flag: '--retry-attempts <0..3>', desc: 'same-pool retries per failed action', default: '1' },
|
|
592
592
|
{ flag: '--resume <shortId|runId>', desc: 'resume a previously started run instead of starting a new goal; mutually exclusive with typing new goal text', default: 'starts a new goal' },
|
|
593
593
|
{ flag: '--detach', desc: 'rarely needed — explicitly requests the default independent-launch behavior; cannot combine with --watch', default: 'the default launch already detaches' },
|
package/src/workflow/cli.js
CHANGED
|
@@ -400,7 +400,9 @@ async function wfGoal(opts) {
|
|
|
400
400
|
if (!opts.foreground && !resumeRunId && !opts.request) {
|
|
401
401
|
const launch = await launchDetachedGoal(doc, opts);
|
|
402
402
|
if (shouldAutoWatchGoal(opts)) {
|
|
403
|
-
|
|
403
|
+
// The detached child writes state.json asynchronously; give it a
|
|
404
|
+
// bounded grace period instead of failing the handoff on a slow host.
|
|
405
|
+
return runWorkflowWatch(BULLSWARM_DIR(), launch.runId ?? launch.shortId, { waitForRunMs: 30_000 });
|
|
404
406
|
}
|
|
405
407
|
return 0;
|
|
406
408
|
}
|
package/src/workflow/decision.js
CHANGED
|
@@ -32,16 +32,39 @@ export function parseDecisionText(text) {
|
|
|
32
32
|
// a verifier depending on exactly one action plainly intends to review that
|
|
33
33
|
// action's durable output artifact. Ambiguous or missing dependencies remain
|
|
34
34
|
// untouched and are rejected by normal validation.
|
|
35
|
+
// verify.review is a dotted scope path to the artifact under review
|
|
36
|
+
// (outputs.<actionId>.outFile), never instructions. Planners routinely put the
|
|
37
|
+
// reviewer's instructions there; recover that shape instead of failing a
|
|
38
|
+
// dispatch: instructions move to prompt, and a single-dependency verify reviews
|
|
39
|
+
// its dependency's artifact.
|
|
40
|
+
export const REVIEW_PATH_RE = /^outputs\.([A-Za-z0-9_-]+(?:\[\d+\])?)\.outFile$/;
|
|
41
|
+
|
|
42
|
+
export function looksLikeReviewPath(value) {
|
|
43
|
+
return typeof value === 'string' && REVIEW_PATH_RE.test(value.trim());
|
|
44
|
+
}
|
|
45
|
+
|
|
35
46
|
export function normalizeDecisionProposal(proposal) {
|
|
36
47
|
if (!proposal || typeof proposal !== 'object' || !Array.isArray(proposal.actions)) return proposal;
|
|
37
48
|
return {
|
|
38
49
|
...proposal,
|
|
39
50
|
actions: proposal.actions.map((action) => {
|
|
40
|
-
if (action?.type !== 'verify'
|
|
41
|
-
|
|
42
|
-
|
|
51
|
+
if (action?.type !== 'verify') return action;
|
|
52
|
+
const singleDependency = Array.isArray(action.dependsOn) && action.dependsOn.length === 1
|
|
53
|
+
? action.dependsOn[0] : null;
|
|
54
|
+
if (action.review == null) {
|
|
55
|
+
return singleDependency ? { ...action, review: `outputs.${singleDependency}.outFile` } : action;
|
|
56
|
+
}
|
|
57
|
+
if (looksLikeReviewPath(action.review)) return { ...action, review: action.review.trim() };
|
|
58
|
+
if (typeof action.review === 'string' && singleDependency) {
|
|
59
|
+
const { review, ...rest } = action;
|
|
60
|
+
return {
|
|
61
|
+
...rest,
|
|
62
|
+
...(rest.prompt == null && /\s/.test(review.trim()) ? { prompt: review } : {}),
|
|
63
|
+
review: `outputs.${singleDependency}.outFile`,
|
|
64
|
+
reviewNormalizedFrom: review,
|
|
65
|
+
};
|
|
43
66
|
}
|
|
44
|
-
return
|
|
67
|
+
return action;
|
|
45
68
|
}),
|
|
46
69
|
};
|
|
47
70
|
}
|
|
@@ -77,6 +100,7 @@ export function validateDecisionProposal(proposal, {
|
|
|
77
100
|
}
|
|
78
101
|
|
|
79
102
|
const known = new Set(knownActionIds);
|
|
103
|
+
const proposedIds = new Set(safeActions.map((action) => action?.id).filter((id) => typeof id === 'string'));
|
|
80
104
|
const closed = new Set(closedPhases);
|
|
81
105
|
const proposed = new Set();
|
|
82
106
|
let proposedItems = 0;
|
|
@@ -122,7 +146,18 @@ export function validateDecisionProposal(proposal, {
|
|
|
122
146
|
}
|
|
123
147
|
}
|
|
124
148
|
}
|
|
125
|
-
if (action.type === 'verify'
|
|
149
|
+
if (action.type === 'verify') {
|
|
150
|
+
if (typeof action.review !== 'string') {
|
|
151
|
+
issues.push(`${at}.review is required: a verify with one dependsOn reviews that artifact automatically; with several, set review to "outputs.<actionId>.outFile" and put reviewer instructions in prompt`);
|
|
152
|
+
} else if (!looksLikeReviewPath(action.review)) {
|
|
153
|
+
issues.push(`${at}.review must be a dotted artifact path like "outputs.<actionId>.outFile" (the artifact to review), not instructions or a filesystem path; put reviewer instructions in ${at}.prompt`);
|
|
154
|
+
} else {
|
|
155
|
+
const reviewed = REVIEW_PATH_RE.exec(action.review.trim())[1].replace(/\[\d+\]$/, '');
|
|
156
|
+
if (!known.has(reviewed) && !proposedIds.has(reviewed)) {
|
|
157
|
+
issues.push(`${at}.review references unknown action "${reviewed}"`);
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
}
|
|
126
161
|
}
|
|
127
162
|
if (proposedItems > maxItemsPerExpansion) {
|
|
128
163
|
issues.push(`proposal has ${proposedItems} fanout items, exceeding maxItemsPerExpansion=${maxItemsPerExpansion}`);
|
package/src/workflow/goal.js
CHANGED
|
@@ -15,10 +15,10 @@ export const AUTONOMOUS_ORCHESTRATOR_PROMPT = [
|
|
|
15
15
|
'',
|
|
16
16
|
'At every checkpoint:',
|
|
17
17
|
'1. Observe the intent, completed actions, artifacts, failures, verification results, available capabilities, and remaining budget.',
|
|
18
|
-
'2. If evidence is insufficient, return needs_more_work with the
|
|
19
|
-
'3. Give workers self-contained prompts with the exact goal,
|
|
20
|
-
'4. Assign every action a short kebab-case phase name such as discover,
|
|
21
|
-
'5. Use
|
|
18
|
+
'2. If evidence is insufficient, return needs_more_work with the COMPLETE dependency graph of bounded run, fanout, and verify actions you can see now — not the smallest step. Independent actions run concurrently; dependent actions start as soon as their dependencies succeed. One planning round trip costs minutes, so a decision with one action when several are obvious is the expensive choice.',
|
|
19
|
+
'3. Give workers self-contained prompts with the exact goal, absolute working directory, the files they may edit (and that they must not touch others), the expected artifact, and the exact acceptance command. A worker sees only its own prompt.',
|
|
20
|
+
'4. Assign every action a short kebab-case phase name such as discover, fix, verify-items, or verify-suite. Phases are forward-only: never append new work to a phase that already finished.',
|
|
21
|
+
'5. Use dependsOn only for real data or same-file ordering dependencies. For N independent items propose N fix actions and N verify actions (each verify depending only on its own fix) plus one final verify depending on all of them; use fanout when every item needs the identical prompt.',
|
|
22
22
|
' A verify action with exactly one dependency automatically reviews that dependency artifact; you do not need to supply a review path.',
|
|
23
23
|
'6. Recover from a failed action with a new bounded action in a new phase when useful; do not repeat an identical failed plan.',
|
|
24
24
|
'7. Require concrete verification of changed behavior. For code changes, obtain relevant test or inspection evidence before completion.',
|
|
@@ -66,7 +66,7 @@ export function buildGoalWorkflow({
|
|
|
66
66
|
const maxActions = positiveInt(settings.maxActions, 40, { max: 1000 });
|
|
67
67
|
const maxItemsPerExpansion = positiveInt(settings.maxItemsPerExpansion, 8, { max: 100 });
|
|
68
68
|
const maxWorkflowSeconds = positiveInt(settings.maxWorkflowSeconds, 3600, { max: 86_400 });
|
|
69
|
-
const concurrency = positiveInt(settings.concurrency,
|
|
69
|
+
const concurrency = positiveInt(settings.concurrency, 8, { max: 16 });
|
|
70
70
|
const retryAttempts = positiveInt(settings.retryAttempts, 1, { min: 0, max: 3 });
|
|
71
71
|
if (!['agent-decides', 'off', 'required'].includes(worktreeIsolation)) {
|
|
72
72
|
throw new Error(`invalid worktree isolation policy "${worktreeIsolation}"`);
|
package/src/workflow/runner.js
CHANGED
|
@@ -546,13 +546,68 @@ async function runDecisionLoop({ runtime, gate, phase, state, retryAttempts }) {
|
|
|
546
546
|
const settings = state.settings;
|
|
547
547
|
const executionPhase = (action) => `${phase}:${action.phase ?? 'adaptive'}`;
|
|
548
548
|
|
|
549
|
+
// Ready-set scheduler. Every action whose dependencies have succeeded is
|
|
550
|
+
// launched immediately; a dependent action starts the moment its own
|
|
551
|
+
// dependencies finish, not when the whole wave finishes. Real concurrency is
|
|
552
|
+
// capped by the runtime's global dispatch limiter (settings.concurrency), so
|
|
553
|
+
// this mirrors a pipeline: item B's verify overlaps item C's fix.
|
|
549
554
|
const executeActions = async (actions) => {
|
|
550
555
|
const pending = new Map(actions.map((action) => [action.id, action]));
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
556
|
+
const running = new Map();
|
|
557
|
+
const runOne = async (action) => {
|
|
558
|
+
if (state.outputs[action.id]?.ok === true) {
|
|
559
|
+
runtime.emit('action.resumed', { actionId: action.id, status: 'succeeded' });
|
|
560
|
+
return;
|
|
561
|
+
}
|
|
562
|
+
let result;
|
|
563
|
+
try {
|
|
564
|
+
state.currentStep = {
|
|
565
|
+
id: action.id,
|
|
566
|
+
type: action.type,
|
|
567
|
+
phase: executionPhase(action),
|
|
568
|
+
};
|
|
569
|
+
runtime.persist();
|
|
570
|
+
result = await runtime.runStep(
|
|
571
|
+
{ ...action, parentId: gate.id, _dynamic: true },
|
|
572
|
+
{ phase: executionPhase(action), retryAttempts },
|
|
573
|
+
);
|
|
574
|
+
} catch (err) {
|
|
575
|
+
// A post-artifact observer failure must not rewrite a durably
|
|
576
|
+
// successful action as failed. Propagate it to the gate boundary;
|
|
577
|
+
// resume will use the persisted output and continue pending work.
|
|
578
|
+
if (state.outputs[action.id]?.ok === true) throw err;
|
|
579
|
+
result = { ok: false, why: err.message };
|
|
580
|
+
state.outputs[action.id] = result;
|
|
581
|
+
runtime.setActionStatus(action, { phase: executionPhase(action) }, 'failed_terminal', {
|
|
582
|
+
finishedAt: new Date().toISOString(), why: err.message,
|
|
583
|
+
});
|
|
584
|
+
runtime.emit('action.failed', { actionId: action.id, status: 'failed_terminal', why: err.message });
|
|
585
|
+
}
|
|
586
|
+
state.steps.push({
|
|
587
|
+
phase: executionPhase(action), stepId: action.id, type: action.type,
|
|
588
|
+
ok: result.ok, why: result.why ?? null, dynamic: true,
|
|
589
|
+
});
|
|
590
|
+
runtime.persist();
|
|
591
|
+
};
|
|
592
|
+
let observerError = null;
|
|
593
|
+
while (pending.size || running.size) {
|
|
594
|
+
if (state.cancelRequested && !running.size) return { ok: false, why: 'workflow cancellation requested' };
|
|
595
|
+
const ready = state.cancelRequested ? [] : [...pending.values()].filter((action) =>
|
|
554
596
|
(action.dependsOn ?? []).every((id) => state.outputs[id]?.ok === true));
|
|
555
|
-
|
|
597
|
+
for (const action of ready) {
|
|
598
|
+
pending.delete(action.id);
|
|
599
|
+
const task = runOne(action)
|
|
600
|
+
.catch((err) => { observerError ??= err; })
|
|
601
|
+
.finally(() => running.delete(action.id));
|
|
602
|
+
running.set(action.id, task);
|
|
603
|
+
}
|
|
604
|
+
if (running.size) {
|
|
605
|
+
await Promise.race([...running.values()]);
|
|
606
|
+
if (observerError) throw observerError;
|
|
607
|
+
continue;
|
|
608
|
+
}
|
|
609
|
+
if (state.cancelRequested) return { ok: false, why: 'workflow cancellation requested' };
|
|
610
|
+
if (pending.size) {
|
|
556
611
|
const blocked = [...pending.values()].map((action) => ({ id: action.id, dependsOn: action.dependsOn ?? [] }));
|
|
557
612
|
const why = `dynamic actions blocked by failed or unresolved dependencies: ${JSON.stringify(blocked)}`;
|
|
558
613
|
for (const action of pending.values()) {
|
|
@@ -567,42 +622,6 @@ async function runDecisionLoop({ runtime, gate, phase, state, retryAttempts }) {
|
|
|
567
622
|
// it can propose a bounded retry/escalation or stop truthfully.
|
|
568
623
|
return { ok: true, blocked, why };
|
|
569
624
|
}
|
|
570
|
-
for (const action of ready) {
|
|
571
|
-
pending.delete(action.id);
|
|
572
|
-
if (state.outputs[action.id]?.ok === true) {
|
|
573
|
-
runtime.emit('action.resumed', { actionId: action.id, status: 'succeeded' });
|
|
574
|
-
continue;
|
|
575
|
-
}
|
|
576
|
-
let result;
|
|
577
|
-
try {
|
|
578
|
-
state.currentStep = {
|
|
579
|
-
id: action.id,
|
|
580
|
-
type: action.type,
|
|
581
|
-
phase: executionPhase(action),
|
|
582
|
-
};
|
|
583
|
-
runtime.persist();
|
|
584
|
-
result = await runtime.runStep(
|
|
585
|
-
{ ...action, parentId: gate.id, _dynamic: true },
|
|
586
|
-
{ phase: executionPhase(action), retryAttempts },
|
|
587
|
-
);
|
|
588
|
-
} catch (err) {
|
|
589
|
-
// A post-artifact observer failure must not rewrite a durably
|
|
590
|
-
// successful action as failed. Propagate it to the gate boundary;
|
|
591
|
-
// resume will use the persisted output and continue pending work.
|
|
592
|
-
if (state.outputs[action.id]?.ok === true) throw err;
|
|
593
|
-
result = { ok: false, why: err.message };
|
|
594
|
-
state.outputs[action.id] = result;
|
|
595
|
-
runtime.setActionStatus(action, { phase: executionPhase(action) }, 'failed_terminal', {
|
|
596
|
-
finishedAt: new Date().toISOString(), why: err.message,
|
|
597
|
-
});
|
|
598
|
-
runtime.emit('action.failed', { actionId: action.id, status: 'failed_terminal', why: err.message });
|
|
599
|
-
}
|
|
600
|
-
state.steps.push({
|
|
601
|
-
phase: executionPhase(action), stepId: action.id, type: action.type,
|
|
602
|
-
ok: result.ok, why: result.why ?? null, dynamic: true,
|
|
603
|
-
});
|
|
604
|
-
runtime.persist();
|
|
605
|
-
}
|
|
606
625
|
}
|
|
607
626
|
return { ok: true };
|
|
608
627
|
};
|
package/src/workflow/runtime.js
CHANGED
|
@@ -987,6 +987,8 @@ export class WorkflowRuntime {
|
|
|
987
987
|
.filter(Boolean))],
|
|
988
988
|
budget,
|
|
989
989
|
executionConstraints: {
|
|
990
|
+
concurrency: Number(this.state.settings?.concurrency ?? 1) || 1,
|
|
991
|
+
readySiblingsRunConcurrently: true,
|
|
990
992
|
actionTimeoutSec: Number(step.actionDefaults?.timeoutSec ?? step.timeoutSec) || null,
|
|
991
993
|
actionTimeoutIsExplicitOptIn: step.actionDefaults?.timeoutSec != null || step.timeoutSec != null,
|
|
992
994
|
retryAttempts: Number(this.state.settings?.retryAttempts ?? 0),
|
|
@@ -1035,19 +1037,34 @@ export class WorkflowRuntime {
|
|
|
1035
1037
|
'Allowed decisions: proceed, complete, needs_more_work, retry, escalate, wait_for_approval, stop.',
|
|
1036
1038
|
'Every proposed action MUST use the field "type" (never "kind").',
|
|
1037
1039
|
'Every action MUST include a forward-only kebab-case "phase". Never reuse a name listed in closedPhases.',
|
|
1040
|
+
'',
|
|
1041
|
+
'PLANNING DOCTRINE — propose the whole graph, not one step:',
|
|
1042
|
+
'- Each planning turn is a full round trip through a separate process (typically 1-2 minutes). Propose the COMPLETE dependency graph you can see now as one decision. A decision carrying a single action when several independent actions are already obvious wastes a round trip.',
|
|
1043
|
+
'- Execution is a ready-set scheduler: every action whose dependsOn have all succeeded starts immediately and concurrently, up to executionConstraints.concurrency at once; a dependent action starts the moment its own dependencies finish, not when the whole round finishes. Independent actions therefore run in parallel automatically. Only add dependsOn where a real data or file-ordering dependency exists.',
|
|
1044
|
+
'- Per-item chains: for N independent items (modules, files, findings), propose N focused run actions plus N verify actions, each verify depending only on its own run, so verifying one item overlaps with fixing another. Add one final verify depending on all of them for the whole-system check.',
|
|
1045
|
+
'- File ownership: every action prompt must name exactly which files it may edit and state that it must not touch any other file. Two actions that must edit the same file MUST be ordered with dependsOn; never let concurrent actions write the same file.',
|
|
1046
|
+
'- Self-contained prompts: a worker sees only its own prompt, never this context. Each prompt must state the absolute working directory, what to read, what to change, the exact command that proves success, and what to report back. Prefer many small parallel actions over one large serial one.',
|
|
1047
|
+
'',
|
|
1038
1048
|
'Action skeletons (copy the shape exactly; every field shown is required unless marked optional):',
|
|
1039
1049
|
' run: {"id":"bounded-action","type":"run","phase":"implement","prompt":"Do bounded work.","dependsOn":["prior-action"]}',
|
|
1040
1050
|
' fanout: {"id":"per-item-check","type":"fanout","phase":"inspect","items":["alpha","beta"],"stepTemplate":{"prompt":"Inspect {{item}} and report concrete evidence."},"dependsOn":["prior-action"]}',
|
|
1041
|
-
' verify: {"id":"independent-check","type":"verify","phase":"verify","
|
|
1051
|
+
' verify: {"id":"independent-check","type":"verify","phase":"verify","prompt":"Independently re-run the tests and report pass/fail with evidence.","dependsOn":["bounded-action"]}',
|
|
1052
|
+
'verify semantics: the reviewer receives the artifact of the action named in review, which the runtime infers as outputs.<the single dependsOn>.outFile; put the reviewer INSTRUCTIONS in prompt. A verify with several dependsOn must set review explicitly to "outputs.<actionId>.outFile". review is never instructions or a filesystem path.',
|
|
1053
|
+
'Graph skeleton (two parallel fix→verify chains plus a final whole-suite check, all in ONE decision):',
|
|
1054
|
+
' [{"id":"fix-alpha","type":"run","phase":"fix","prompt":"In /abs/repo edit only src/alpha.js so tests/alpha.test.js passes; run node --test tests/alpha.test.js; report the diff summary."},',
|
|
1055
|
+
' {"id":"fix-beta","type":"run","phase":"fix","prompt":"In /abs/repo edit only src/beta.js so tests/beta.test.js passes; run node --test tests/beta.test.js; report the diff summary."},',
|
|
1056
|
+
' {"id":"verify-alpha","type":"verify","phase":"verify-items","prompt":"Re-run node --test tests/alpha.test.js in /abs/repo and confirm tests/ is unchanged.","dependsOn":["fix-alpha"]},',
|
|
1057
|
+
' {"id":"verify-beta","type":"verify","phase":"verify-items","prompt":"Re-run node --test tests/beta.test.js in /abs/repo and confirm tests/ is unchanged.","dependsOn":["fix-beta"]},',
|
|
1058
|
+
' {"id":"verify-suite","type":"verify","phase":"verify-suite","prompt":"Run the full npm test in /abs/repo and report pass/fail counts.","review":"outputs.fix-beta.outFile","dependsOn":["verify-alpha","verify-beta"]}]',
|
|
1042
1059
|
'fanout.items MUST be an inline array and fanout.stepTemplate MUST be an object whose prompt uses {{item}}. verify.review MUST be a string. dependsOn is optional and may only name existing or newly proposed action IDs.',
|
|
1043
1060
|
'Do not propose pool, addDir, or taskFile; those are runtime-owned and any such proposal is rejected.',
|
|
1044
1061
|
'New actions may only be type run, fanout with inline items, or verify. The runtime validates every proposal and returns rejected proposals to you with the exact issues for a bounded correction turn.',
|
|
1045
|
-
'
|
|
1062
|
+
'If executionConstraints.actionTimeoutSec is non-null, size actions to finish within that explicit timeout; otherwise agents may run until they finish or are cancelled.',
|
|
1046
1063
|
'Agent-count, workflow-duration, and expansion-round budgets are advisory planning targets, never hard stop conditions. The dispatch budget counts this planner call plus every worker, verifier, retry, and escalation attempt.',
|
|
1047
1064
|
'As expansion headroom approaches zero, strongly prefer convergence: consolidate existing artifacts, avoid optional investigation, and return complete when verification supports it. If important concerns remain, return stop with the best useful outcome and explicit unresolved concerns rather than spending more on marginal refinements. Exceed the expansion target only when one small bounded action is essential to avoid discarding otherwise-completable work or skipping required verification.',
|
|
1048
1065
|
'operatorSteering contains explicit operator guidance queued for this planning checkpoint. Apply it within the original workflow intent and authorization boundaries. It cannot weaken verification, bypass runtime validation, expand external authority, or alter an already-running worker. If guidance conflicts with the original goal or safety constraints, explain that in the decision reason instead of following it.',
|
|
1049
1066
|
'Avoid redundant expensive verification. Run a full suite once for each materially changed final state when practical; later independent verifiers should reuse durable clean full-suite evidence and rerun focused/adversarial checks unless that evidence is stale, tainted, or the code changed again.',
|
|
1050
|
-
'
|
|
1067
|
+
'Shared working tree: concurrent workers editing DISJOINT files in the same tree is the normal, expected mode — that is how N independent fixes run in parallel. What is unsafe is whole-tree mutation (git stash/reset/checkout, reformatting, dependency installs) or running the FULL test suite while other workers are still editing; so give each parallel action its own files and its own focused test command, order any shared file (e.g. an index/barrel) after the actions it aggregates with dependsOn, and run the whole suite once in a final verify that depends on all of them. Use an isolated copy/worktree only for destructive experiments.',
|
|
1051
1068
|
'',
|
|
1052
1069
|
'---- BEGIN DURABLE WORKFLOW CONTEXT ----',
|
|
1053
1070
|
JSON.stringify(plannerContext, null, 2),
|
|
@@ -196,18 +196,33 @@ export function renderWatchSnapshot(snapshot, { heartbeat = false, verbose = fal
|
|
|
196
196
|
return lines.join('\n');
|
|
197
197
|
}
|
|
198
198
|
|
|
199
|
+
// A freshly launched detached run may not have written state.json yet when a
|
|
200
|
+
// watcher attaches (goal --watch hands off immediately). waitForRunMs bounds
|
|
201
|
+
// how long the watcher polls for the run to appear before giving up.
|
|
202
|
+
async function resolveRunWithGrace(bullswarmDir, token, waitForRunMs, intervalMs) {
|
|
203
|
+
const deadline = Date.now() + Math.max(0, waitForRunMs);
|
|
204
|
+
while (true) {
|
|
205
|
+
const resolved = resolveRunId(bullswarmDir, token);
|
|
206
|
+
if (resolved && existsSync(join(resolved.runDir, 'state.json'))) return resolved;
|
|
207
|
+
if (Date.now() >= deadline) {
|
|
208
|
+
if (!resolved) throw new Error(`no run found for "${token}"`);
|
|
209
|
+
throw new Error(`run "${token}" has no state.json`);
|
|
210
|
+
}
|
|
211
|
+
await new Promise((resolve) => setTimeout(resolve, Math.min(250, Math.max(50, intervalMs))));
|
|
212
|
+
}
|
|
213
|
+
}
|
|
214
|
+
|
|
199
215
|
export async function runWorkflowWatch(bullswarmDir, token, {
|
|
200
216
|
intervalMs = 2000,
|
|
201
217
|
heartbeatMs = 60000,
|
|
202
218
|
once = false,
|
|
203
219
|
jsonl = false,
|
|
204
220
|
verbose = false,
|
|
221
|
+
waitForRunMs = 0,
|
|
205
222
|
output = process.stdout,
|
|
206
223
|
} = {}) {
|
|
207
|
-
const resolved =
|
|
208
|
-
if (!resolved) throw new Error(`no run found for "${token}"`);
|
|
224
|
+
const resolved = await resolveRunWithGrace(bullswarmDir, token, waitForRunMs, intervalMs);
|
|
209
225
|
const statePath = join(resolved.runDir, 'state.json');
|
|
210
|
-
if (!existsSync(statePath)) throw new Error(`run "${token}" has no state.json`);
|
|
211
226
|
let priorFingerprint = null;
|
|
212
227
|
let priorHumanFingerprint = null;
|
|
213
228
|
let lastPrintedAt = 0;
|