bullswarm 0.25.4 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +7 -7
- package/CHANGELOG.md +179 -0
- package/README.md +142 -28
- package/connectors/codex.json +3 -2
- package/connectors/command-code.json +229 -27
- package/data/README.md +139 -9
- package/data/epoch-benchmarks.json +6028 -0
- package/docs/studies/portal-token-diet.md +127 -0
- package/package.json +4 -2
- package/skill/SKILL.md +27 -34
- package/skill/references/operations.md +107 -3
- package/src/cli.js +111 -8
- package/src/help.js +89 -52
- package/src/integrate.js +4 -11
- package/src/lib/assignments.js +332 -0
- package/src/lib/epoch-benchmarks.js +394 -0
- package/src/lib/forecast.js +111 -0
- package/src/lib/route.js +363 -67
- package/src/lib/spend.js +452 -0
- package/src/lib/strategy.js +209 -2
- package/src/meters/framework.js +51 -0
- package/src/meters/registry.js +135 -2
- package/src/setup.js +76 -24
- package/src/strategy-cli.js +215 -3
- package/src/strategy-dashboard.js +5 -2
- package/src/workflow/action-validator.js +117 -4
- package/src/workflow/cli.js +72 -4
- package/src/workflow/runs-cli.js +34 -0
- package/src/workflow/runtime.js +40 -1
- package/src/workflow/v2-dispatch.js +60 -3
- package/src/workflow/v2-outcome.js +8 -1
- package/src/workflow/v2-planner.js +31 -5
- package/src/workflow/v2-runtime.js +3 -3
- package/src/workflow/v2-state.js +21 -2
- package/src/delegate.js +0 -438
package/AGENTS.md
CHANGED
|
@@ -59,13 +59,13 @@ bullswarm workflow runs delete <shortId> --yes
|
|
|
59
59
|
## Using bullswarm from another agent
|
|
60
60
|
|
|
61
61
|
If you are an agent that wants to offload bounded work via bullswarm,
|
|
62
|
-
read `skill/SKILL.md` — that's the agent-facing user guide.
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
`workflow
|
|
67
|
-
|
|
68
|
-
|
|
62
|
+
read `skill/SKILL.md` — that's the agent-facing user guide. There are
|
|
63
|
+
exactly two ways to start work, and the caller chooses the shape itself: one
|
|
64
|
+
bounded outcome goes to `bullswarm run`; parallel territories, integration,
|
|
65
|
+
or independent acceptance go to `bullswarm workflow goal` with a program you
|
|
66
|
+
author (`bullswarm workflow plan contract` returns the schema). There is no
|
|
67
|
+
classifier or preview step. The skill is published alongside the package and
|
|
68
|
+
is the canonical reference for the CLI surface.
|
|
69
69
|
|
|
70
70
|
- Zero runtime dependencies. Node >= 18. Tests must never require network:
|
|
71
71
|
prime `~/.bullswarm/meters/*.json` caches with fresh timestamps if needed.
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,184 @@
|
|
|
1
1
|
# bullswarm changelog
|
|
2
2
|
|
|
3
|
+
## 0.26.0 — two entry points, kinds and rungs
|
|
4
|
+
|
|
5
|
+
- There are now exactly two ways to start work, and `bullswarm delegate` is
|
|
6
|
+
gone. `delegate` existed to decide, on the caller's behalf, whether a request
|
|
7
|
+
needed one agent or a workflow — with `--mode auto` it spent a real
|
|
8
|
+
analyze-lane dispatch on a deterministic-then-LLM classifier before doing any
|
|
9
|
+
of the actual work. The calling agent already knows the shape of its own
|
|
10
|
+
request, so that round trip bought latency and quota, not accuracy. `bullswarm
|
|
11
|
+
run` is the single-agent entry point and `bullswarm workflow goal` is the
|
|
12
|
+
workflow entry point; `bullswarm --help` names those two, in that order, and
|
|
13
|
+
`bullswarm delegate` now exits 2 with the standard unknown-verb message and
|
|
14
|
+
dispatches nothing. The packaged skill and the awareness block registered into
|
|
15
|
+
Codex, Claude, and Grok say the same thing in six lines: decide the shape
|
|
16
|
+
yourself, there is no preview step.
|
|
17
|
+
|
|
18
|
+
- `bullswarm run --lane analyze` now defaults to `medium` effort instead of
|
|
19
|
+
`high`. The V2 validator has always defaulted an analyze action to medium, so
|
|
20
|
+
the same lane meant a different tier depending on which entry point you used.
|
|
21
|
+
`run` and the validator now read one exported `DEFAULT_EFFORT_BY_LANE` table,
|
|
22
|
+
and `run --help` states the corrected default.
|
|
23
|
+
|
|
24
|
+
- Program actions can now say what they *are* instead of restating how to route
|
|
25
|
+
them. The optional `kind` field takes one of seven values — `mechanical`,
|
|
26
|
+
`io-read`, `check`, `implement`, `integration`, `architecture`,
|
|
27
|
+
`adversarial-acceptance` — and derives `lane` and `effort` from a table
|
|
28
|
+
exported alongside the existing per-lane default. Resolution is per field: an
|
|
29
|
+
explicit action field wins, then the kind table, then a new optional
|
|
30
|
+
program-level `defaults` object (`effort` and `reasoning` only), then the lane
|
|
31
|
+
default. A kind outside the closed list is a validation error with the allowed
|
|
32
|
+
values named, because it is a typo in the program rather than a runtime
|
|
33
|
+
condition. Programs that use neither `kind` nor `defaults` and state
|
|
34
|
+
`lane` and `effort` on every action normalise byte-identically to before;
|
|
35
|
+
`effort` itself is now optional and falls back to the per-lane default
|
|
36
|
+
where it used to be rejected as missing.
|
|
37
|
+
|
|
38
|
+
- Two non-blocking advisories, never rejections. `all-writers-high` fires when
|
|
39
|
+
three or more build/chore actions run and none is below high effort;
|
|
40
|
+
`docs-at-high` fires when a build/chore action owns only `*.md` files at high
|
|
41
|
+
effort. `workflow plan validate --json` carries them as `advisories` and the
|
|
42
|
+
human output prints `advisory:` lines; `workflow goal --program` prints the
|
|
43
|
+
same lines at launch. Neither changes acceptance or an exit code. The kernel
|
|
44
|
+
records them on the run, and `workflow runs show` lists them.
|
|
45
|
+
|
|
46
|
+
- `workflow runs result`, `runs show`, and `action show` print `kind` next to
|
|
47
|
+
lane and effort when an action has one. `workflow action show` now understands
|
|
48
|
+
autonomous V2 runs at all — it previously read only the V1 action ledger and
|
|
49
|
+
failed on every V2 run.
|
|
50
|
+
|
|
51
|
+
- **Rungs: one pool's model plus its reasoning level, for one effort tier, read
|
|
52
|
+
and written as one thing.** `bullswarm strategy rungs [--json] [--pool <name>]`
|
|
53
|
+
prints one row per enabled pool and configured tier with the effective model
|
|
54
|
+
and its source, the effective reasoning level and the layer that chose it, the
|
|
55
|
+
dated Epoch benchmark evidence for that model *at that level* (`blended`, cost
|
|
56
|
+
per task, tokens per task), and the local record from the decision log for that
|
|
57
|
+
pool and tier (dispatches, median wall minutes, ok share). Absent evidence
|
|
58
|
+
prints `no evidence` and an unmeasured tier prints `no dispatches`; neither is
|
|
59
|
+
ever estimated. `strategy inventory --json` gained the same rows under `rungs`.
|
|
60
|
+
|
|
61
|
+
- `bullswarm strategy set-rung <pool> <tier> --model <model> [--reasoning <level>]
|
|
62
|
+
[--force]` writes both halves of a rung in one atomic state save, so the model
|
|
63
|
+
and the thinking depth can never land separately. A rung is singular per pool
|
|
64
|
+
and tier: the tier moves off whichever model held it, and that model keeps its
|
|
65
|
+
other tiers. A level the connector cannot express is clamped to the strongest
|
|
66
|
+
it accepts and the clamp is printed. An unknown pool or tier exits 2; a model
|
|
67
|
+
absent from the pool's cached discovery exits 2 and lists the known models
|
|
68
|
+
unless `--force` is given. Neither `rungs` nor `set-rung` ever spawns model
|
|
69
|
+
discovery. No state migration: rungs are a projection of `strategy.modelTiers`
|
|
70
|
+
and `strategy.reasoning`, and `~/.bullswarm/state.json` gained no keys.
|
|
71
|
+
|
|
72
|
+
- The setup wizard's tier step now shows each suggested rung with its benchmark
|
|
73
|
+
evidence line and asks one reasoning question per configured tier. **Behavior
|
|
74
|
+
change:** Enter keeps that connector's own per-tier default and writes nothing,
|
|
75
|
+
where the previous question stored a suggested level (`xhigh`/`high`/`medium`)
|
|
76
|
+
on a blank answer. Connector defaults remain the final fallback, so a pool with
|
|
77
|
+
no configured rung behaves exactly as before. Non-TTY and `--yes` paths are
|
|
78
|
+
unchanged.
|
|
79
|
+
|
|
80
|
+
- A dated evidence datapack per model *and reasoning level*, from Epoch AI.
|
|
81
|
+
`data/epoch-benchmarks.json` (schema `bullswarm.epoch.benchmarks.v1`) is built
|
|
82
|
+
by the new `scripts/refresh-epoch-benchmarks.mjs` from Epoch's cursorbench,
|
|
83
|
+
deepswe, arc-agi-2, and critpt exports, and `src/lib/epoch-benchmarks.js`
|
|
84
|
+
reads it with the same cache → bundled → URL fallback as the OpenRouter pack.
|
|
85
|
+
`rungEvidence()` returns the mean of whichever of those four scores exist for a
|
|
86
|
+
(model, level) pair as `blended`, with cost and tokens per task from
|
|
87
|
+
cursorbench; `normalizeModelId()` is how connector model ids match the export.
|
|
88
|
+
The data is used under CC BY 4.0 — Epoch AI, 'AI Benchmarking Hub'. Published
|
|
89
|
+
online at epoch.ai. Retrieved from https://epoch.ai/benchmarks.
|
|
90
|
+
|
|
91
|
+
- The daily refresh job is renamed `.github/workflows/refresh-benchmarks.yml` and
|
|
92
|
+
now refreshes both assets on the rolling `benchmark-data-latest` release: it
|
|
93
|
+
downloads and unzips Epoch's public export, runs the script, runs the new
|
|
94
|
+
tests, and uploads `epoch-benchmarks.json` next to `openrouter-benchmarks.json`.
|
|
95
|
+
The ambiguous `npm run refresh:benchmarks` script is split into
|
|
96
|
+
`refresh:openrouter` and `refresh:epoch`, since only one of the two now
|
|
97
|
+
refreshes "the benchmarks".
|
|
98
|
+
|
|
99
|
+
- The OpenRouter builder is unchanged, and that is a finding rather than an
|
|
100
|
+
omission: the Artificial Analysis records in the 2026-09-08 capture carry only
|
|
101
|
+
`agentic_index`, `coding_index`, and `intelligence_index` per model, with no
|
|
102
|
+
reasoning-effort marker on any of the `reasoning_effort`, `effort`, `variant`,
|
|
103
|
+
`reasoning`, or `reasoning_level` fields checked, so there is no per-effort row
|
|
104
|
+
to keep. `data/README.md` records the field names inspected.
|
|
105
|
+
|
|
106
|
+
## 0.25.5 — forecast-aware routing
|
|
107
|
+
|
|
108
|
+
- Bullswarm now knows what it is already running. Every dispatch registers the
|
|
109
|
+
work it starts in a small ledger on disk (`~/.bullswarm/assignments/`, one
|
|
110
|
+
atomically written file per assignment), and every process reads it: a
|
|
111
|
+
`bullswarm run` in one terminal, a V1 runtime and four concurrent V2 kernel
|
|
112
|
+
actions all see each other's agents instead of each assuming the pool is
|
|
113
|
+
idle. Records whose process is gone, or that are older than 12 hours, are
|
|
114
|
+
pruned on read, so a crash cannot leave phantom load behind. `bullswarm
|
|
115
|
+
assignments` lists what is in flight right now — pool, run, action, how long
|
|
116
|
+
it has been going and how much longer it is expected to take — and
|
|
117
|
+
`bullswarm pools` carries the same count as `inflight=<n>`.
|
|
118
|
+
|
|
119
|
+
- A spend model turns those records into percentage points. It measures how
|
|
120
|
+
fast a pool actually burns its 5-hour and weekly windows by pairing meter
|
|
121
|
+
readings with the worker-minutes dispatched between them, and how long an
|
|
122
|
+
assignment on a given lane and effort tier usually runs by taking the median
|
|
123
|
+
of real attempts from the decision log. Every number carries its basis —
|
|
124
|
+
`history` (measured), `bootstrap` (one window's usage so far), or the
|
|
125
|
+
documented `default` table — and the sample count behind it. A pool nobody
|
|
126
|
+
has measured reports `null`, never a plausible-looking guess.
|
|
127
|
+
|
|
128
|
+
- Routing now decides on the forecast instead of on the last reading. Each
|
|
129
|
+
pool's projection (reading + what its in-flight agents will still spend) gets
|
|
130
|
+
the expected consumption of the assignment being routed added on top, and the
|
|
131
|
+
existing thresholds apply to that number: a pool projected at or above 75% of
|
|
132
|
+
its 5-hour window drops to the near-limit tier while its reading is still
|
|
133
|
+
below the line, and one projected at or above 90% is dropped from selection
|
|
134
|
+
as forecast-gated. Nothing is gated on an unknown forecast, and if every
|
|
135
|
+
capable pool is gated, the least loaded of them is still picked — with the
|
|
136
|
+
reason saying exactly that — rather than the action being stranded.
|
|
137
|
+
|
|
138
|
+
- Parallel work now spreads instead of stacking. Within a tier, a pool's pace
|
|
139
|
+
surplus is reduced by the weekly quota its in-flight agents and this
|
|
140
|
+
assignment are expected to spend, and by at least a flat 3 surplus points
|
|
141
|
+
per in-flight agent (`DEFAULT_INFLIGHT_PENALTY_PCT`, `config.inflightPenaltyPct`
|
|
142
|
+
in state.json, `0` disables it). The floor matters: at measured weekly rates
|
|
143
|
+
a six-minute agent projects to under one point, which would leave a burst on
|
|
144
|
+
one pool. The charge is labeled `penalty` when the floor set it and by its
|
|
145
|
+
measured basis otherwise. Load also beats incumbency: an incumbent carrying
|
|
146
|
+
more in-flight agents than a challenger keeps neither its margin nor its cost
|
|
147
|
+
guard. Four actions launched within the same second land on
|
|
148
|
+
four different providers rather than all on the single most-behind one, and
|
|
149
|
+
the V2 kernel re-reads the ledger before every pick rather than only on its
|
|
150
|
+
throttled meter refresh, so actions launched seconds apart still see each
|
|
151
|
+
other.
|
|
152
|
+
|
|
153
|
+
- Everything that observes routing shows the new numbers. `bullswarm
|
|
154
|
+
assignments [--json]` is a new command listing the live ledger; `bullswarm
|
|
155
|
+
pools` gained an `inflight=<n>` column and prints its 5-hour cell as
|
|
156
|
+
`5h=<reading>%-><projected>%` when in-flight work is expected to move it,
|
|
157
|
+
with `spend`, `projectedFiveHourPct` and `projectedWeeklyPct` in `--json`;
|
|
158
|
+
`bullswarm run --dry-run` prints the forecast the pick was made on and, being
|
|
159
|
+
a preview, still registers nothing and writes no decision log; and the
|
|
160
|
+
strategy control center shows each provider's in-flight count next to its
|
|
161
|
+
usage. Live meter readings are now retained as a capped per-pool series at
|
|
162
|
+
`~/.bullswarm/meters/history/<pool>.jsonl`, because the snapshot cache keeps
|
|
163
|
+
only the newest reading and a rate needs two.
|
|
164
|
+
|
|
165
|
+
- A rate is only reported once the dispatch behind it is real: at least five
|
|
166
|
+
worker-minutes must be attributable to a window before its utilization
|
|
167
|
+
counts as percentage-points-per-minute. Without that floor a pool at 26% of
|
|
168
|
+
its 5-hour window with one six-second-old agent measures as 260% per minute
|
|
169
|
+
and forecasts every provider past the burst line, which is the failure this
|
|
170
|
+
model exists to prevent rather than cause.
|
|
171
|
+
|
|
172
|
+
- All of it is visible after the fact. The routing reason names the in-flight
|
|
173
|
+
counts and projections that moved the pick (`5h used 30% -> 41% projected, 2
|
|
174
|
+
in flight`, `skipped near 5h limit (projected): wati 76%`, `forecast-gated
|
|
175
|
+
at/above 90%: …`, `preferred over busier: …`), every candidate row carries
|
|
176
|
+
`pace`, `effectiveSurplus`, `inflight`, `projectedFiveHourPct`,
|
|
177
|
+
`forecastFiveHourPct`, `projectedWeeklyPct`, `ratePerMinute`,
|
|
178
|
+
`estimateSource` and `forecastGated`, and the decision log records the
|
|
179
|
+
forecast the pick was made on. Pools that carry no ledger or spend fields
|
|
180
|
+
route exactly as they did before.
|
|
181
|
+
|
|
3
182
|
## 0.25.4 — reasoning levels
|
|
4
183
|
|
|
5
184
|
- A connector now declares how its own CLI expresses a thinking level, and
|
package/README.md
CHANGED
|
@@ -6,10 +6,24 @@ evidence agents by quota, and computes completion from a durable requirement
|
|
|
6
6
|
ledger without an initiating agent authoring a graph.
|
|
7
7
|
Every delegate output is judged by content before it counts.
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
9
|
+
## Entry points
|
|
10
|
+
|
|
11
|
+
There are exactly two ways to start work:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
bullswarm run --lane analyze --add-dir ~/some-repo --prompt "Explain the parser" --json
|
|
15
|
+
bullswarm workflow goal "Fix the failing tests and verify the change" --cwd ~/some-repo --program plan.json
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
`bullswarm run` is the single-agent entry point: route, dispatch, watch,
|
|
19
|
+
verify, one JSON verdict. `bullswarm workflow goal` is the workflow entry
|
|
20
|
+
point: you author the bounded action program and the kernel executes it to the
|
|
21
|
+
end. For agents, `/bullswarm` (or `$bullswarm` where skills use that syntax)
|
|
22
|
+
reads the packaged skill and goes straight to one of those two. The calling
|
|
23
|
+
agent decides the shape itself from the request — one bounded outcome takes
|
|
24
|
+
`run`, parallel territories, integration, or independent acceptance take
|
|
25
|
+
`workflow goal`. There is no preview, classifier, or dispatcher command
|
|
26
|
+
between them.
|
|
13
27
|
|
|
14
28
|
Every command and nested subcommand supports contextual `-h` / `--help`
|
|
15
29
|
without initializing state or executing the command:
|
|
@@ -79,9 +93,6 @@ bullswarm setup # interactive provider/model configuration
|
|
|
79
93
|
bullswarm setup --wizard # broader worktree + integration questionnaire
|
|
80
94
|
bullswarm pools # meter state, pace position, quarantine status
|
|
81
95
|
bullswarm strategy # explicit alias for the same routing control center
|
|
82
|
-
bullswarm delegate --cwd ~/some-repo --prompt "Explain the parser" # one agent
|
|
83
|
-
bullswarm delegate --cwd ~/some-repo --prompt "Audit all commands, fix help, and independently verify" # workflow
|
|
84
|
-
bullswarm delegate --dry-run --json --cwd ~/some-repo --prompt "Your task" # bounded classification + decision/plan; no work dispatch
|
|
85
96
|
bullswarm run --lane analyze --add-dir ~/some-repo --task-file /tmp/t.md --json
|
|
86
97
|
bullswarm run --lane analyze --add-dir ~/some-repo --prompt "Inspect the parser" --json
|
|
87
98
|
bullswarm workflow plan contract "Fix the failing tests and verify the change" --cwd ~/some-repo --json # you are the planner
|
|
@@ -96,7 +107,6 @@ bullswarm health # re-judge saved outputs; catch gate failures
|
|
|
96
107
|
|---|---|
|
|
97
108
|
| `setup` | Discover installed agent CLIs, show quota state, toggle pools, suggest a routing table, write config. Approval-gated, idempotent. |
|
|
98
109
|
| `integrate` | Register or remove the canonical Bullswarm skill and global awareness rules for Codex, Claude, and Grok. |
|
|
99
|
-
| `delegate` | Explain and execute the smallest reliable shape: one content-verified agent, or the planning contract for an autonomous workflow you author (`--orchestrator` dispatches a planner agent instead). |
|
|
100
110
|
| `run` | route → dispatch → watch → verify → one JSON verdict |
|
|
101
111
|
| `health` | Re-judge saved outputs against their verdicts; surface verify-gate failures and quarantine clusters |
|
|
102
112
|
| `pools` | Show each pool's meter state, pace position, 5-hour utilization (`5h=<n>%`, flagged `NEAR-5H-LIMIT` at or above 75%), quarantine status |
|
|
@@ -107,22 +117,6 @@ bullswarm health # re-judge saved outputs; catch gate failures
|
|
|
107
117
|
| `version` / `--version` | Print the installed Bullswarm version. |
|
|
108
118
|
| `release` | Run the guarded local version-bump, commit, and tag workflow used before CI publishes to npm. |
|
|
109
119
|
|
|
110
|
-
### Delegate classification
|
|
111
|
-
|
|
112
|
-
With the default `--mode auto`, `delegate` first uses deterministic task
|
|
113
|
-
signals, then uses an LLM to refine the choice between a single delegate and a
|
|
114
|
-
workflow during execution. If that optional refinement is unavailable or
|
|
115
|
-
unusable, automatic mode uses the deterministic decision.
|
|
116
|
-
|
|
117
|
-
Use `--classify deterministic` to bypass the LLM refinement — this is the
|
|
118
|
-
instant, no-dispatch preview. Use `--classify llm` when an LLM decision is
|
|
119
|
-
required: the command fails if it cannot obtain a usable one. In automatic
|
|
120
|
-
mode, `--dry-run` still performs that same bounded low-effort classification
|
|
121
|
-
request (one analyze-lane, low-effort dispatch) and prints the resulting
|
|
122
|
-
decision — it never dispatches the work itself. An explicit `--mode single` or
|
|
123
|
-
`--mode workflow` is the caller's decision and bypasses automatic LLM
|
|
124
|
-
classification.
|
|
125
|
-
|
|
126
120
|
Discover and validate workflow definitions without executing them:
|
|
127
121
|
|
|
128
122
|
```bash
|
|
@@ -163,6 +157,44 @@ bullswarm strategy exclude-model claude-fable-5
|
|
|
163
157
|
bullswarm run --effort high --lane analyze --task-file /tmp/task.md --json
|
|
164
158
|
```
|
|
165
159
|
|
|
160
|
+
### Rungs
|
|
161
|
+
|
|
162
|
+
A **rung** is one pool's model *plus its reasoning level* for one effort tier —
|
|
163
|
+
the two halves you actually choose together. `bullswarm strategy rungs` reads
|
|
164
|
+
them as one table and `bullswarm strategy set-rung` writes both halves in one
|
|
165
|
+
atomic save:
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
bullswarm strategy rungs # every enabled pool x configured tier
|
|
169
|
+
bullswarm strategy rungs --json --pool codex # machine-readable, one pool
|
|
170
|
+
bullswarm strategy set-rung codex high --model gpt-5.6-sol --reasoning xhigh
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Each row carries the effective model and where it came from, the effective
|
|
174
|
+
reasoning level and which layer chose it, the dated benchmark evidence for that
|
|
175
|
+
model *at that reasoning level* (`blended`, `$/task`, `tok/task`), and what this
|
|
176
|
+
machine recorded for that pool and tier (dispatch count, median wall minutes, ok
|
|
177
|
+
share). Evidence and record are never estimated: a model the datapack does not
|
|
178
|
+
cover prints `no evidence`, and a tier with no matching attempt prints `no
|
|
179
|
+
dispatches`. `strategy inventory --json` carries the identical rows under
|
|
180
|
+
`rungs`.
|
|
181
|
+
|
|
182
|
+
Reading is free of side effects — no state write, no model discovery, no
|
|
183
|
+
download. `set-rung` never spawns discovery either: a model absent from the
|
|
184
|
+
pool's cached discovery exits 2 and lists the models it does know, unless you
|
|
185
|
+
pass `--force`. A reasoning level the connector cannot express is clamped down
|
|
186
|
+
to the strongest level it accepts and the clamp is printed. A rung is singular
|
|
187
|
+
per pool and tier, so the tier moves off whichever model held it while that
|
|
188
|
+
model keeps its other tiers. Nothing about `state.json` changed shape: rungs are
|
|
189
|
+
a view over `strategy.modelTiers` and `strategy.reasoning`.
|
|
190
|
+
|
|
191
|
+
The benchmark evidence comes from Epoch AI's benchmarking hub, used under
|
|
192
|
+
CC BY 4.0: Epoch AI, 'AI Benchmarking Hub'. Published online at epoch.ai.
|
|
193
|
+
Retrieved from <https://epoch.ai/benchmarks>. `blended` is the mean of the
|
|
194
|
+
cursorbench, deepswe, arc-agi-2, and critpt scores recorded for that exact
|
|
195
|
+
model and reasoning level; cost and tokens per task come from cursorbench. See
|
|
196
|
+
[data/README.md](data/README.md) for the schema and the refresh job.
|
|
197
|
+
|
|
166
198
|
Setup first asks whether to analyze live usage and recommend routes or open the
|
|
167
199
|
current configuration for manual editing. Analysis shows a spinner plus
|
|
168
200
|
per-provider usage progress, then presents the proposed defaults before making
|
|
@@ -170,10 +202,13 @@ any routing change. Press `Y` to apply them or `N` to retain the current policy.
|
|
|
170
202
|
The analysis selects at most one default model for each provider and effort
|
|
171
203
|
tier. It uses OpenRouter's agentic, coding, and intelligence indices as quality
|
|
172
204
|
signals and API-equivalent pricing as the budget signal. A repository-owned
|
|
173
|
-
GitHub Actions job
|
|
174
|
-
|
|
175
|
-
`
|
|
176
|
-
|
|
205
|
+
GitHub Actions job (`.github/workflows/refresh-benchmarks.yml`) refreshes two
|
|
206
|
+
public assets on the rolling `benchmark-data-latest` GitHub Release:
|
|
207
|
+
`openrouter-benchmarks.json` from the authenticated OpenRouter APIs, and
|
|
208
|
+
`epoch-benchmarks.json` from Epoch AI's CC BY 4.0 benchmark export, which is
|
|
209
|
+
what `strategy rungs` reads for per-model-per-reasoning-level evidence.
|
|
210
|
+
Installed CLIs download only those public files and never need or receive an
|
|
211
|
+
OpenRouter key.
|
|
177
212
|
The sources are OpenRouter's [benchmarks API](https://openrouter.ai/docs/api/api-reference/benchmarks/list-benchmarks)
|
|
178
213
|
and [models API](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties).
|
|
179
214
|
The CLI caches the datapack under `~/.bullswarm/cache/`; network failure falls
|
|
@@ -227,6 +262,49 @@ name the utilization that decided the pick, and meters and quarantines are
|
|
|
227
262
|
re-read before each dispatch — and again, live, right after a usage limit —
|
|
228
263
|
so a long run never routes off the snapshot it launched with.
|
|
229
264
|
|
|
265
|
+
Those thresholds are applied to the FORECAST, not to the last reading. A meter
|
|
266
|
+
reading is already old when it arrives: agents dispatched seconds ago have
|
|
267
|
+
spent quota the provider has not reported yet, and the assignment being routed
|
|
268
|
+
will spend more. So each pool's projection — its reading plus the quota its
|
|
269
|
+
in-flight agents are still expected to burn — gets this candidate's own
|
|
270
|
+
expected consumption added, and the tiers apply to that number: a pool
|
|
271
|
+
projected at or above 75% drops to the near-limit tier even while its reading
|
|
272
|
+
is lower, and one projected at or above `BURST_BLOCK_PCT` (90) is left out of
|
|
273
|
+
selection entirely as forecast-gated. If every capable pool is forecast-gated,
|
|
274
|
+
routing still names the least loaded of them rather than stranding the action,
|
|
275
|
+
and says so in the reason. A pool with no measured rate forecasts nothing and
|
|
276
|
+
is never gated or deprioritized for a number nobody produced.
|
|
277
|
+
|
|
278
|
+
Within a tier, pools already carrying work yield to quieter pools of similar
|
|
279
|
+
pace: each pool's surplus is reduced by the weekly quota its in-flight agents
|
|
280
|
+
and this assignment are expected to spend, and by at least a flat 3 surplus
|
|
281
|
+
points per in-flight agent. That floor is what spreads work at real rates,
|
|
282
|
+
where a six-minute agent projects to well under one point; the charge is
|
|
283
|
+
labeled `penalty` when the floor set it and carries its measured basis
|
|
284
|
+
(`history`, `bootstrap`) when the projection was larger. Load also beats
|
|
285
|
+
incumbency: an incumbent carrying more in-flight agents than a challenger keeps
|
|
286
|
+
neither its 10-point margin nor its cost protection, so the quieter pool wins
|
|
287
|
+
as soon as its effective surplus is higher. A burst of parallel actions
|
|
288
|
+
therefore spreads across providers instead of stacking on the single
|
|
289
|
+
most-behind one.
|
|
290
|
+
`bullswarm pools` shows each pool's `inflight=<n>` count and its 5-hour column
|
|
291
|
+
as `5h=<reading>%-><projected>%` whenever in-flight work is expected to move
|
|
292
|
+
it, `bullswarm assignments` lists what those agents are, `bullswarm run
|
|
293
|
+
--dry-run` prints the forecast the pick was made on without registering
|
|
294
|
+
anything, and every candidate row carries `pace`, `effectiveSurplus`,
|
|
295
|
+
`inflight`, `projectedFiveHourPct`, `forecastFiveHourPct`, `ratePerMinute`,
|
|
296
|
+
`estimateSource` and `forecastGated`, so a surprising pick can be read back
|
|
297
|
+
number by number.
|
|
298
|
+
|
|
299
|
+
The rates come from real records: every live meter reading is retained as a
|
|
300
|
+
capped per-pool series (`~/.bullswarm/meters/history/<pool>.jsonl`) and paired
|
|
301
|
+
with the worker-minutes dispatched between readings. Until at least five
|
|
302
|
+
worker-minutes of dispatch are attributable to a window there is no rate at
|
|
303
|
+
all — `null`, not a ratio of percentage points to seconds — so a fresh machine
|
|
304
|
+
routes on pace and the flat penalty until it has measured something. The
|
|
305
|
+
penalty itself is `config.inflightPenaltyPct` in `~/.bullswarm/state.json`
|
|
306
|
+
(default 3; `0` turns the tie-breaker off).
|
|
307
|
+
|
|
230
308
|
Model exclusions are hard routing policy. An excluded model is removed from
|
|
231
309
|
recommendations and assignments, and Bullswarm pins a same-tier allowed model
|
|
232
310
|
through the connector-owned model flag whenever the provider default could be
|
|
@@ -305,6 +383,42 @@ adversarial acceptance judgment. Merely being an analysis/evidence action or
|
|
|
305
383
|
part of a difficult goal never promotes an action to high. The selected effort
|
|
306
384
|
then resolves through the High/Medium/Low routes configured by `bullswarm setup`.
|
|
307
385
|
|
|
386
|
+
### Kinds
|
|
387
|
+
|
|
388
|
+
Stating lane and effort separately on every action means re-deciding two
|
|
389
|
+
fields for work whose nature already implies both. The optional `kind` field
|
|
390
|
+
names that nature once and derives them:
|
|
391
|
+
|
|
392
|
+
| `kind` | lane | effort |
|
|
393
|
+
| --- | --- | --- |
|
|
394
|
+
| `mechanical` | chore | low |
|
|
395
|
+
| `io-read` | analyze | low |
|
|
396
|
+
| `check` | analyze | medium |
|
|
397
|
+
| `implement` | build | medium |
|
|
398
|
+
| `integration` | build | high |
|
|
399
|
+
| `architecture` | analyze | high |
|
|
400
|
+
| `adversarial-acceptance` | analyze | high |
|
|
401
|
+
|
|
402
|
+
Resolution is per field: an explicit `lane` or `effort` on the action wins,
|
|
403
|
+
then the kind table, then an optional program-level `defaults` object — which
|
|
404
|
+
may set only `effort` and `reasoning`, because lane follows the individual
|
|
405
|
+
action — then the per-lane default table. A `kind` outside that closed list is
|
|
406
|
+
a validation error, not a runtime failure: it is a typo in your program, so
|
|
407
|
+
`workflow plan validate` exits 2 and nothing launches. A program that uses
|
|
408
|
+
neither `kind` nor `defaults` and states `lane` and `effort` on every action
|
|
409
|
+
validates and runs exactly as before; the one widening is that `effort` is now
|
|
410
|
+
optional and falls back to the per-lane default instead of being rejected.
|
|
411
|
+
|
|
412
|
+
Two advisories report effort smells without ever rejecting anything.
|
|
413
|
+
`all-writers-high` fires when three or more `build`/`chore` actions run and
|
|
414
|
+
none is below high effort; `docs-at-high` fires when a `build`/`chore` action
|
|
415
|
+
owns only `*.md` files at high effort. `workflow plan validate` includes them
|
|
416
|
+
as `advisories` in `--json` and prints `advisory:` lines otherwise, `workflow
|
|
417
|
+
goal` prints the same lines at launch, and both keep their exit codes. The
|
|
418
|
+
kernel stores them on the run, so `workflow runs show` lists them afterwards,
|
|
419
|
+
and `runs result`, `runs show`, and `workflow action show` print `kind` next to
|
|
420
|
+
lane and effort.
|
|
421
|
+
|
|
308
422
|
Reasoning depth is a third, independent decision. An action may carry an
|
|
309
423
|
optional `reasoning` field — `low`, `medium`, `high`, `xhigh`, `max`, or
|
|
310
424
|
`default` — that sets how hard the picked model thinks on that one action and
|
package/connectors/codex.json
CHANGED
|
@@ -117,7 +117,7 @@
|
|
|
117
117
|
"flag": "--model",
|
|
118
118
|
"mode": "replace-or-append"
|
|
119
119
|
},
|
|
120
|
-
"$comment-reasoning": "
|
|
120
|
+
"$comment-reasoning": "verified 2026-09-09 against codex-cli 0.153.4: `codex exec -c model_reasoning_effort=max --model gpt-5.6-luna` ran (exit 0), printed `reasoning effort: max`, and the session rollout recorded reasoning_effort \"max\"; the public config reference still lists only minimal|low|medium|high|xhigh. `minimal` is accepted by the CLI but sits below bullswarm's ladder and is not declared.",
|
|
121
121
|
"reasoning": {
|
|
122
122
|
"args": [
|
|
123
123
|
"-c",
|
|
@@ -127,7 +127,8 @@
|
|
|
127
127
|
"low",
|
|
128
128
|
"medium",
|
|
129
129
|
"high",
|
|
130
|
-
"xhigh"
|
|
130
|
+
"xhigh",
|
|
131
|
+
"max"
|
|
131
132
|
],
|
|
132
133
|
"defaults": {
|
|
133
134
|
"high": "high",
|