@robmclarty/volley 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +42 -0
- package/README.md +398 -38
- package/dist/cli.js +4469 -2930
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +260 -36
- package/dist/index.js +1296 -450
- package/dist/index.js.map +1 -1
- package/package.json +3 -2
- package/src/builder/presets/harness_append_acp.md +86 -0
- package/src/builder/presets/harness_append_local.md +2 -1
- package/src/critic/presets/harness_append.md +2 -1
- package/src/critic/presets/harness_append_acp.md +27 -0
- package/src/critic/presets/harness_append_local.md +4 -3
package/README.md
CHANGED
|
@@ -54,8 +54,9 @@ volley \
|
|
|
54
54
|
The cap is enforced in the loop guard after each phase. Sessions are never
|
|
55
55
|
killed mid-flight for cost (a builder killed mid-edit leaves the workspace
|
|
56
56
|
undefined), so set the cap below your true ceiling by roughly one iteration's
|
|
57
|
-
expected cost. Per-run totals land in
|
|
58
|
-
terminal summary.
|
|
57
|
+
expected cost. Per-run totals land in the run's `summary.json` (see
|
|
58
|
+
[Run artifacts](#run-artifacts)) and the final terminal summary. An [OpenRouter](#openrouter) seat's cost is the charge
|
|
59
|
+
OpenRouter reports for each call, so the cap trips on one as on any other.
|
|
59
60
|
|
|
60
61
|
## How it works
|
|
61
62
|
|
|
@@ -80,7 +81,7 @@ prompt ──> builder (full agentic session in your workspace)
|
|
|
80
81
|
- **Critic** is read-only (`Read`, `Grep`, `Glob` — no write path at all) and
|
|
81
82
|
returns a schema-validated `{ verdict, feedback, unmet_criteria }`. It is told
|
|
82
83
|
which files the builder changed, so it reviews a change rather than a tree.
|
|
83
|
-
The harness writes
|
|
84
|
+
The harness writes the run's `feedback.md` and `verdict`; the critic
|
|
84
85
|
cannot get them wrong.
|
|
85
86
|
|
|
86
87
|
Each iteration is a fresh session — no context carries over except through the
|
|
@@ -94,7 +95,7 @@ Under the hood the loop is one [fascicle](https://www.npmjs.com/package/fascicle
|
|
|
94
95
|
composition in [`src/flow.ts`](./src/flow.ts), and fascicle's `describe.diagram`
|
|
95
96
|
draws it straight from the code. It shows what the sketch above leaves out: the
|
|
96
97
|
branch that skips verification once the builder alone crosses the cost cap, the
|
|
97
|
-
critic's retry-then-tool-less fallback
|
|
98
|
+
critic's retry-then-tool-less fallback on every provider but `claude_cli`, and
|
|
98
99
|
the guard that decides when the loop stops.
|
|
99
100
|
|
|
100
101
|
<!-- flow diagram: pnpm diagram -->
|
|
@@ -108,8 +109,8 @@ volley loop: one round per iteration, up to max_i
|
|
|
108
109
|
│ │ └─ else sequence
|
|
109
110
|
│ │ ├─ check the deterministic gate: checkride, a command, or none
|
|
110
111
|
│ │ └─ critique the read-only critic's verdict and feedback
|
|
111
|
-
│ │ └─ critic fallback: judge without tools if a
|
|
112
|
-
│ │ ├─ retry one more try after a stream death,
|
|
112
|
+
│ │ └─ critic fallback: judge without tools if a critic on any provider but claude_cli keeps dying or runs out of steps
|
|
113
|
+
│ │ ├─ retry one more try after a stream death, on any provider but claude_cli
|
|
113
114
|
│ │ │ └─ critic_tools model call with read-only workspace tools
|
|
114
115
|
│ │ └─ pipe mark the verdict degraded
|
|
115
116
|
│ │ └─ critic_toolless model call judging from the check output and a file list
|
|
@@ -122,8 +123,10 @@ the one in the `src/flow.ts` header to what the code builds, so neither drifts.
|
|
|
122
123
|
|
|
123
124
|
## Install
|
|
124
125
|
|
|
125
|
-
Requires Node ≥ 24
|
|
126
|
-
installed and
|
|
126
|
+
Requires Node ≥ 24 and pnpm. The default seats run on the
|
|
127
|
+
[Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code), installed and
|
|
128
|
+
authenticated; seats on [OpenRouter](#openrouter) or a
|
|
129
|
+
[local model server](#local-critic) don't need it.
|
|
127
130
|
|
|
128
131
|
From a checkout:
|
|
129
132
|
|
|
@@ -147,6 +150,9 @@ volley [options]
|
|
|
147
150
|
volley --config <path>
|
|
148
151
|
volley resume <run-id>
|
|
149
152
|
volley matrix --builders <models> --critics <models> --config <path>
|
|
153
|
+
volley critique --criteria <criteria> [--baseline <rev>] [options]
|
|
154
|
+
volley acp --builder-provider <name> --builder-model <model> \
|
|
155
|
+
--critic-provider <name> --critic-model <model> [options]
|
|
150
156
|
```
|
|
151
157
|
|
|
152
158
|
| Flag | Default | Description |
|
|
@@ -156,13 +162,14 @@ volley matrix --builders <models> --critics <models> --config <path>
|
|
|
156
162
|
| `--criteria` | (required) | Acceptance criteria. Literal string or `@path`. |
|
|
157
163
|
| `--check` | `auto` | `auto` (checkride if detected, else none), `none`, or a shell command. |
|
|
158
164
|
| `--builder-model` | `opus` | `opus`/`sonnet`/`haiku` or a full model id. |
|
|
159
|
-
| `--builder-provider` | `claude_cli` | `claude_cli`, `ollama`, or `
|
|
160
|
-
| `--builder-url` | the provider's env var | The builder's model server base URL, for `ollama` or `
|
|
161
|
-
| `--builder-max-steps` | `50` |
|
|
162
|
-
| `--
|
|
165
|
+
| `--builder-provider` | `claude_cli` | `claude_cli`, `ollama`, `lmstudio`, or `openrouter` (see [Local builder](#local-builder) and [OpenRouter](#openrouter)). |
|
|
166
|
+
| `--builder-url` | the provider's env var | The builder's model server base URL, for `ollama`, `lmstudio`, or `openrouter` only. Lets the builder and critic run on different machines with the same provider (see [Seats on different machines](#seats-on-different-machines)). |
|
|
167
|
+
| `--builder-max-steps` | `50` | Tool-loop builder step cap per iteration. Or `VOLLEY_BUILDER_MAX_STEPS`. Ignored for `claude_cli`. |
|
|
168
|
+
| `--builder-max-tool-calls` | `1` | Tool calls a tool-loop builder may make per turn; past it, fascicle runs the first ones and drops the rest. Raise it for a model and runtime that return parallel calls well. Or `VOLLEY_BUILDER_MAX_TOOL_CALLS`. Ignored for `claude_cli`. |
|
|
169
|
+
| `--allow-unsandboxed-builder` | off | Permit a builder other than `claude_cli` to run without a sandbox (see [Local builder](#local-builder)). |
|
|
163
170
|
| `--critic-model` | `opus` | Model for the critic. |
|
|
164
|
-
| `--critic-provider` | `claude_cli` | `claude_cli`, `ollama`, or `
|
|
165
|
-
| `--critic-url` | the provider's env var | The critic's model server base URL, for `ollama` or `
|
|
171
|
+
| `--critic-provider` | `claude_cli` | `claude_cli`, `ollama`, `lmstudio`, or `openrouter` (see [Local critic](#local-critic) and [OpenRouter](#openrouter)). |
|
|
172
|
+
| `--critic-url` | the provider's env var | The critic's model server base URL, for `ollama`, `lmstudio`, or `openrouter` only. |
|
|
166
173
|
| `--builder-permission-mode` | `acceptEdits` | Or `bypassPermissions` for fully trusted workspaces. |
|
|
167
174
|
| `--critic` | `reviewer` | `reviewer`, `optimizer`, `researcher`, or a prompt file path. |
|
|
168
175
|
| `--max-iterations` | `10` | Hard iteration cap. |
|
|
@@ -173,8 +180,9 @@ volley matrix --builders <models> --critics <models> --config <path>
|
|
|
173
180
|
| `--gate-paths` | (built-in list) | Comma-separated globs naming the gate — tests, fixtures, check config. Replaces the defaults; see [What the builder changed](#what-the-builder-changed). |
|
|
174
181
|
| `--fail-on-gate-edit` | off | Halt the run (exit 8) if the builder edits a gate path, instead of reporting it. |
|
|
175
182
|
| `--sandbox-image` | `volley-sandbox:latest` | Container image for the local-builder sandbox. Or `VOLLEY_SANDBOX_IMAGE`. |
|
|
176
|
-
| `--
|
|
177
|
-
| `--
|
|
183
|
+
| `--state-dir` | `$XDG_STATE_HOME/volley`, else `~/.local/state/volley` | The root each run keeps its state under, at `<path>/<run id>/`, never in the workspace (see [Run artifacts](#run-artifacts)). Or `state_dir` in a config file. |
|
|
184
|
+
| `--run-id` | a fresh UUID | The run's id, naming its run state and its `volley/<id>` branch, so a harness that owns the id finds it in both. Letters, digits, `-`, and `_`, starting with a letter or digit. |
|
|
185
|
+
| `--dry-run` | off | Validate config, run `checkride doctor`, probe each tool-loop seat's model endpoint (local or OpenRouter), run the critic-seat canary (for a local critic), then exit. |
|
|
178
186
|
| `--config` | — | TypeScript config file (`VolleyConfig` default export). CLI flags override. |
|
|
179
187
|
| `--json` | off | Machine mode: one summary JSON document on stdout. |
|
|
180
188
|
| `--verbose` | off | Full tool inputs/outputs (truncated at 4000 chars). |
|
|
@@ -190,7 +198,7 @@ checkride.
|
|
|
190
198
|
| Code | Meaning |
|
|
191
199
|
|---|---|
|
|
192
200
|
| 0 | Check passed and verdict approved. |
|
|
193
|
-
| 2 | Max iterations reached without success. |
|
|
201
|
+
| 2 | Max iterations reached without success; for `volley critique`, the critic asked for changes or the check failed. |
|
|
194
202
|
| 3 | Builder/provider error. |
|
|
195
203
|
| 4 | Check pipeline error (broken pipeline, not failing checks). |
|
|
196
204
|
| 5 | Configuration error. |
|
|
@@ -223,12 +231,35 @@ export default config;
|
|
|
223
231
|
### Resume
|
|
224
232
|
|
|
225
233
|
```sh
|
|
226
|
-
volley resume <run-id>
|
|
227
|
-
volley resume <run-id> --
|
|
234
|
+
volley resume <run-id> # from any directory
|
|
235
|
+
volley resume <run-id> --state-dir ~/volley-runs # a run started with --state-dir
|
|
236
|
+
volley resume <run-id> --workspace ./my-project # a run an older volley kept in .volley/
|
|
228
237
|
```
|
|
229
238
|
|
|
239
|
+
`resume` finds the run at `<root>/<run-id>/` under `--state-dir`, else the
|
|
240
|
+
default root, from any directory, since the run's recorded config names its
|
|
241
|
+
workspace. A run an older volley started keeps its state in the workspace's
|
|
242
|
+
`.volley/`, and resumes there: point `--workspace` at it, or run `resume` from
|
|
243
|
+
inside it. A run directory with a symbolic link in it is refused, since volley
|
|
244
|
+
never writes one.
|
|
245
|
+
|
|
246
|
+
**A resume on the host runs only a config this machine signed.** A resume runs
|
|
247
|
+
what the run's `config.json` says, its check command and its seats included.
|
|
248
|
+
So volley on the host signs each config it writes, as `config.sig`, under a
|
|
249
|
+
key that stays on the machine (`$XDG_CONFIG_HOME/volley/run-key`, else
|
|
250
|
+
`~/.config/volley/run-key`), and a resume on the host refuses a config whose
|
|
251
|
+
signature doesn't hold (exit 5). That covers:
|
|
252
|
+
|
|
253
|
+
- **A contained run's config.** Nothing is signed inside the sandbox, and the
|
|
254
|
+
builder there can rewrite its own run's config, so resume it inside its
|
|
255
|
+
container, with the same mount. Inside the container, nothing is checked.
|
|
256
|
+
- **Another machine's config, an older volley's, or one you edited.** Read it,
|
|
257
|
+
then pass `--trust-config`, and the resume says which check and seats it's
|
|
258
|
+
about to run before it runs them.
|
|
259
|
+
|
|
230
260
|
Continues from the last completed iteration with the saved feedback and cost
|
|
231
|
-
totals. An iteration interrupted mid-phase is discarded and re-run.
|
|
261
|
+
totals. An iteration interrupted mid-phase is discarded and re-run. A
|
|
262
|
+
[critique](#critique-a-fixed-change) has one round and nothing to resume.
|
|
232
263
|
|
|
233
264
|
### Matrix sweep
|
|
234
265
|
|
|
@@ -261,16 +292,17 @@ local providers share one GPU, so parallel combos would thrash the model loader.
|
|
|
261
292
|
Each run forces `--worktree --discard-worktree` for a clean per-run reset — a
|
|
262
293
|
sweep wants verdicts, so each seat's effects are isolated and then thrown away,
|
|
263
294
|
leaving neither a branch nor a commit behind — so the **workspace must be a git
|
|
264
|
-
repository**.
|
|
265
|
-
|
|
266
|
-
`.volley-matrix/
|
|
295
|
+
repository**. Each attempt keeps its run state under the state root like any
|
|
296
|
+
run, and its summary, which names that state as `run_state_dir`, is copied to
|
|
297
|
+
`.volley-matrix/<builder>__<critic>/run-NN/summary.json`. The whole sweep goes
|
|
298
|
+
to `.volley-matrix/matrix.json`, and one aggregate table goes to stderr:
|
|
267
299
|
|
|
268
300
|
```
|
|
269
|
-
builder critic pass iters wall
|
|
270
|
-
────────────── ───────────── ──── ─────
|
|
271
|
-
qwen3.6:latest qwen3:8b 3/3 1.0 403.4s
|
|
272
|
-
qwen3.6:latest gemma4:12b 2/3 1.5 512.0s
|
|
273
|
-
qwen3.6:latest glm-4.7-flash 0/3 — 184.0s
|
|
301
|
+
builder critic pass iters wall cost salvage flags why
|
|
302
|
+
────────────── ───────────── ──── ───── ────── ───── ─────── ────────── ──────────────────
|
|
303
|
+
qwen3.6:latest qwen3:8b 3/3 1.0 403.4s $0.00 0% — —
|
|
304
|
+
qwen3.6:latest gemma4:12b 2/3 1.5 512.0s $0.00 0% — unmet: 2 criteria
|
|
305
|
+
qwen3.6:latest glm-4.7-flash 0/3 — 184.0s $0.00 12% deg:stream check: types, test
|
|
274
306
|
```
|
|
275
307
|
|
|
276
308
|
**Why `--repeat`.** One run answers "did this pairing converge that time", and
|
|
@@ -288,7 +320,9 @@ totals rather than a mean of ratios.
|
|
|
288
320
|
failing check slots, the criteria the critic still judged unmet, a cost cap, a
|
|
289
321
|
gate edit, or the error that broke the run. `flags` carries what qualifies a
|
|
290
322
|
pass rather than explaining a failure: `deg` for a degraded critic verdict,
|
|
291
|
-
|
|
323
|
+
named for what degraded it (`deg:stream` when the critic's stream kept dying,
|
|
324
|
+
`deg:steps` when it ran out of steps), and `gate` for a builder that edited the
|
|
325
|
+
gate. Every attempt's full detail —
|
|
292
326
|
statuses, verdicts, failing slots, unmet criteria, gate edits — is in the
|
|
293
327
|
`--json` aggregate and in `matrix.json`.
|
|
294
328
|
|
|
@@ -296,6 +330,48 @@ A run that does not converge is a *result*, shown in the table; the sweep exits
|
|
|
296
330
|
0 as long as every attempt produced a summary, and nonzero only when one
|
|
297
331
|
produced none at all (`status: broke`).
|
|
298
332
|
|
|
333
|
+
### Critique a fixed change
|
|
334
|
+
|
|
335
|
+
`volley critique` runs the check and the critic once over a change already
|
|
336
|
+
made, with no builder. It's how you choose a critic: replay the same
|
|
337
|
+
known-good and known-bad changes through the critic seat, across models and
|
|
338
|
+
prompts, and count how many of the bad ones each critic rejects. A matrix sweep
|
|
339
|
+
can't tell a critic that reads from one that approves everything, since both
|
|
340
|
+
pass whenever the builder happens to be right.
|
|
341
|
+
|
|
342
|
+
```sh
|
|
343
|
+
volley critique --workspace ./repo --baseline <rev> \
|
|
344
|
+
--criteria "@criteria.md" \
|
|
345
|
+
--critic-provider ollama --critic-model qwen3:8b --critic reviewer
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
- **The change** is everything that differs from `--baseline`, committed or
|
|
349
|
+
not, untracked files included: the set a run reports for its builder. Without
|
|
350
|
+
`--baseline` it's HEAD, so the uncommitted and untracked edits are the
|
|
351
|
+
change. A revision that isn't a commit is a configuration error (exit 5).
|
|
352
|
+
- **The critic** sees what it sees in a run: the changed paths and gate edits,
|
|
353
|
+
the check's section, and the same preset or prompt file. `--prompt` is
|
|
354
|
+
optional and only recorded, since a critic judges against `--criteria`.
|
|
355
|
+
- **The state** is a one-round run's, under the state root
|
|
356
|
+
([Run artifacts](#run-artifacts)). The summary records `mode: "critique"`
|
|
357
|
+
and no builder: `builder` is null in the iteration's archive, and so is the
|
|
358
|
+
comparison block's `builder_transport`. The archived `critic.tool_calls` says
|
|
359
|
+
how much the critic read before it judged.
|
|
360
|
+
- **Exit codes**: 0 when the critic approves over a green check, 2 when it asks
|
|
361
|
+
for changes or the check is red (`status: "rejected"`), and the usual codes
|
|
362
|
+
for provider, check-pipeline, configuration, and critic errors.
|
|
363
|
+
- **Flags**: the critic's seat flags, `--check`, `--critic`, `--gate-paths`,
|
|
364
|
+
`--state-dir`, `--run-id`, `--dry-run` (with the critic-seat canary),
|
|
365
|
+
`--json`, and the output flags work as they do for a run. Every flag only a
|
|
366
|
+
builder or its loop uses, from `--builder-model` and `--max-iterations` to
|
|
367
|
+
`--worktree`, `--git`, and `--fail-on-gate-edit`, is refused by name (exit
|
|
368
|
+
5), and so is `--config`, since a run's config file sets a builder.
|
|
369
|
+
|
|
370
|
+
With no builder, a local critic needs no sandbox opt-in. The check still runs
|
|
371
|
+
whatever the repository's check runs, though, so review an untrusted change
|
|
372
|
+
with `volley critique` inside the container, its run directory mounted as a
|
|
373
|
+
run's would be ([Local builder](#local-builder)).
|
|
374
|
+
|
|
299
375
|
## Local critic
|
|
300
376
|
|
|
301
377
|
The critic is read-only and schema-constrained, which makes it a good fit for
|
|
@@ -325,7 +401,7 @@ Providers and their setup:
|
|
|
325
401
|
Install the peer for your provider in the project running volley (e.g.
|
|
326
402
|
`pnpm add ai-sdk-ollama@^4.3.0`); fascicle loads it lazily only when the local
|
|
327
403
|
critic actually runs. The version matters. The major must satisfy fascicle's
|
|
328
|
-
declared peer range (`^4` for fascicle 0.
|
|
404
|
+
declared peer range (`^4` for fascicle 0.16.3, which tracks the AI SDK v7
|
|
329
405
|
line), and volley declares 4.3.0 as its floor (an optional peer, so your
|
|
330
406
|
package manager flags an older copy at install time). Before 4.3.0,
|
|
331
407
|
ai-sdk-ollama streams a thinking model's reasoning only when the request sets
|
|
@@ -356,7 +432,14 @@ the *same* model drives the builder tool loop fine, so the defect is specific to
|
|
|
356
432
|
the constrained-verdict path. volley no longer lets that kill a run: the critic
|
|
357
433
|
phase degrades (a bounded retry, then a tool-less fallback that keeps the
|
|
358
434
|
schema), marks the verdict `critic_degraded: true`, and `--dry-run` predicts the
|
|
359
|
-
doomed combo up front with a critic-seat canary.
|
|
435
|
+
doomed combo up front with a critic-seat canary. A critic that runs out of steps
|
|
436
|
+
before its verdict (its 10-step budget spent, or its last turn a tool call) skips
|
|
437
|
+
the retry, since another identical tool loop would only run out again, and goes
|
|
438
|
+
straight to the tool-less pass. The record's `degrade_cause` says which happened:
|
|
439
|
+
`stream_error` or `step_exhaustion`. A verdict cut off by the token limit or
|
|
440
|
+
blocked by a content filter isn't degraded: it fails the critic phase (exit 6)
|
|
441
|
+
with the finish reason in the message, because taking the tools away fixes
|
|
442
|
+
neither. Even so, prefer a critic whose
|
|
360
443
|
tool-call encoding Ollama parses cleanly — `qwen3:8b`, `gemma4:12b`, and
|
|
361
444
|
`glm-4.7-flash` all pass. The full model-vs-transport write-up is in
|
|
362
445
|
[`research/v3-comparison-finding.md`](./research/v3-comparison-finding.md).
|
|
@@ -371,6 +454,14 @@ tool loop per iteration (capped by `--builder-max-steps`, default 50; hitting
|
|
|
371
454
|
the cap is treated as partial work, not an error). The produced workspace goes
|
|
372
455
|
to check + critic exactly like a `claude_cli` build.
|
|
373
456
|
|
|
457
|
+
A local builder makes one tool call per turn by default, since some model and
|
|
458
|
+
runtime pairs return parallel calls badly. For a pair that handles them, such
|
|
459
|
+
as qwen3.8 on Ollama's MLX runner, which returns several reads at once,
|
|
460
|
+
`--builder-max-tool-calls <n>` lets each turn carry up to `n`. fascicle runs a
|
|
461
|
+
turn's calls in order, so a read can't race an edit, and the builder's prompt
|
|
462
|
+
then asks it to batch independent reads but make one change or command per
|
|
463
|
+
turn.
|
|
464
|
+
|
|
374
465
|
```sh
|
|
375
466
|
volley \
|
|
376
467
|
--prompt "@task.md" --workspace ./my-project --criteria "@criteria.md" \
|
|
@@ -397,6 +488,26 @@ mangled crossing the boundary — so a *contained* Claude role must drive by API
|
|
|
397
488
|
key (`VOLLEY_AUTH_MODE=api_key` with `ANTHROPIC_API_KEY`); the all-Claude
|
|
398
489
|
subscription path stays on the host, Docker-free.
|
|
399
490
|
|
|
491
|
+
**A contained run's state needs a mount of its own.** The container's root
|
|
492
|
+
filesystem is read-only, so volley inside has nowhere to keep the run's state
|
|
493
|
+
until you mount the run's own host directory where it looks:
|
|
494
|
+
|
|
495
|
+
1. Create `<root>/<run id>/` on the host first, empty. On Linux, Docker creates
|
|
496
|
+
a missing one owned by root, and the container's user can't write it.
|
|
497
|
+
2. Mount it at `/home/node/.local/state/volley/<run id>`. That's
|
|
498
|
+
`SANDBOX_STATE_ROOT`, and `run_state` on `sandbox_run_args` renders the
|
|
499
|
+
flag.
|
|
500
|
+
3. Pass volley the same `--run-id`.
|
|
501
|
+
|
|
502
|
+
Without the mount, the run stops before any spend (exit 5) and says what to
|
|
503
|
+
mount. Mount that one directory, never the whole root: the builder runs
|
|
504
|
+
arbitrary commands, and with the root mounted it could read every other run's
|
|
505
|
+
trajectory or plant a check in another run's config. Its own run's config is
|
|
506
|
+
still the builder's to rewrite, so a resume on the host refuses it, since
|
|
507
|
+
volley signs only the configs it writes outside the sandbox
|
|
508
|
+
([Resume](#resume)). Resume a contained run inside the container, with the same
|
|
509
|
+
mount.
|
|
510
|
+
|
|
400
511
|
To run a local builder uncontained on the host anyway, opt out with
|
|
401
512
|
`--allow-unsandboxed-builder` (or `VOLLEY_ALLOW_UNSANDBOXED_BUILDER=1`), which
|
|
402
513
|
prints a one-time warning — only point it at a workspace you are willing to let a
|
|
@@ -494,6 +605,227 @@ on the host, for example `ssh -N -L 11522:localhost:11434 gpu-box`. On a Linux
|
|
|
494
605
|
host, `host-gateway` is the Docker bridge's address, so the forward, like Ollama
|
|
495
606
|
itself, has to listen there rather than on `127.0.0.1` alone.
|
|
496
607
|
|
|
608
|
+
## OpenRouter
|
|
609
|
+
|
|
610
|
+
`--builder-provider openrouter` and `--critic-provider openrouter` put a seat on
|
|
611
|
+
[OpenRouter](https://openrouter.ai), so one API key runs both seats, Claude
|
|
612
|
+
included, with no Claude Code CLI and no local model server:
|
|
613
|
+
|
|
614
|
+
```sh
|
|
615
|
+
OPENROUTER_API_KEY=sk-or-v1-... volley \
|
|
616
|
+
--prompt "@task.md" --workspace ./my-project --criteria "@criteria.md" \
|
|
617
|
+
--builder-provider openrouter --builder-model anthropic/claude-sonnet-5.5 \
|
|
618
|
+
--critic-provider openrouter --critic-model anthropic/claude-haiku-5.5 \
|
|
619
|
+
--allow-unsandboxed-builder --max-iterations 3
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
- **The key.** `OPENROUTER_API_KEY` is required whenever either seat is on
|
|
623
|
+
`openrouter`. Unset, volley exits 5 naming it, before any work. It goes out
|
|
624
|
+
as a Bearer token, with `X-Title: volley` so OpenRouter attributes the
|
|
625
|
+
traffic.
|
|
626
|
+
- **The URL.** `VOLLEY_OPENROUTER_URL` (default `https://openrouter.ai/api/v1`)
|
|
627
|
+
points both seats at a proxy or another OpenAI-compatible gateway, and
|
|
628
|
+
`--builder-url` / `--critic-url` give one seat a URL of its own, under the
|
|
629
|
+
rules in [Seats on different machines](#seats-on-different-machines).
|
|
630
|
+
OpenRouter runs on fascicle's native transport, so there's no peer to
|
|
631
|
+
install, and the run summary records the transport as `native`.
|
|
632
|
+
- **The seats.** An OpenRouter seat runs volley's tool loop, the same as a
|
|
633
|
+
local one. The builder gets volley's tools, `bash` included, so it's refused
|
|
634
|
+
uncontained unless you opt out, exactly like a
|
|
635
|
+
[local builder](#local-builder). The critic gets the read-only tools and the
|
|
636
|
+
same retry-and-degrade ladder as a [local critic](#local-critic).
|
|
637
|
+
`--dry-run` probes each OpenRouter seat's endpoint but runs no critic canary,
|
|
638
|
+
which would cost money here.
|
|
639
|
+
- **Cost and reasoning.** OpenRouter reports what each call cost, and that
|
|
640
|
+
charge is the seat's cost, recorded as `provider_reported`, so
|
|
641
|
+
`--max-cost-usd` trips on an OpenRouter seat as on any other. A model that
|
|
642
|
+
reasons streams its reasoning, which the terminal shows dimmed unless
|
|
643
|
+
`--no-thinking` hides it, and which Zed shows as a thought.
|
|
644
|
+
- **In the sandbox.** The container's allowlist reaches only the host gateway,
|
|
645
|
+
so a contained OpenRouter seat needs `openrouter.ai` let through your
|
|
646
|
+
`DOCKER-USER` rules. volley doesn't install that rule for you.
|
|
647
|
+
|
|
648
|
+
## Run volley from Zed
|
|
649
|
+
|
|
650
|
+
`volley acp` serves volley to an editor over the
|
|
651
|
+
[Agent Client Protocol](https://agentclientprotocol.com), through fascicle's
|
|
652
|
+
`serve_acp`, so a prompt in [Zed](https://zed.dev)'s agent panel starts a volley
|
|
653
|
+
run on the folder Zed has open. The builder's work streams in as the reply, each
|
|
654
|
+
round opens with a **Round N** line, the check shows as a command card, the
|
|
655
|
+
critic's verdict arrives as a thought, and the turn ends with a summary.
|
|
656
|
+
|
|
657
|
+
```sh
|
|
658
|
+
volley acp \
|
|
659
|
+
--builder-provider openrouter --builder-model anthropic/claude-sonnet-5.5 \
|
|
660
|
+
--critic-provider openrouter --critic-model anthropic/claude-haiku-5.5 \
|
|
661
|
+
--check "pnpm test"
|
|
662
|
+
```
|
|
663
|
+
|
|
664
|
+
It reads the protocol on stdin and writes nothing but the protocol to stdout,
|
|
665
|
+
with its own progress on stderr, which Zed shows in its agent log. Zed starts it
|
|
666
|
+
for you ([below](#zed-setup)).
|
|
667
|
+
|
|
668
|
+
### Each prompt is a run
|
|
669
|
+
|
|
670
|
+
- **Fresh, in place.** Each prompt is a new volley run on the open folder, with
|
|
671
|
+
no worktree. Its task and its acceptance criteria are both the conversation so
|
|
672
|
+
far, then the new prompt. An earlier reply keeps only its last 2,000
|
|
673
|
+
characters, where its summary sits. An earlier prompt's attached files shrink
|
|
674
|
+
to a line naming each, since they're still on disk. A turn you stopped is
|
|
675
|
+
marked as stopped.
|
|
676
|
+
- **Three rounds.** `--max-iterations` defaults to 3 per prompt, not 10, since
|
|
677
|
+
a prompt is one short exchange, and `--max-cost-usd` caps what it spends.
|
|
678
|
+
Follow a run that didn't converge with "keep going": the summary it ended
|
|
679
|
+
with lists what the critic still found missing.
|
|
680
|
+
- **The tools.** The builder gets fascicle's workspace tools (`read_file`,
|
|
681
|
+
`list_files`, `search`, `write_file`, `edit_file`, `run_command`) plus
|
|
682
|
+
volley's `finish`. The critic gets the three reads. Both stay inside the open
|
|
683
|
+
folder, and neither gets `fetch`. volley doesn't offer the editor additional
|
|
684
|
+
directories, so in a workspace with several folders Zed opens the session on
|
|
685
|
+
the first, and a client that sends more anyway has its session refused.
|
|
686
|
+
- **The summary.** The turn's last message says whether the run converged, in
|
|
687
|
+
how many rounds, and with what last verdict. When it didn't converge, it
|
|
688
|
+
lists the unmet criteria, and it names any gate edits and any verdict the
|
|
689
|
+
critic reached without its tools. A run that fails ends with a summary saying
|
|
690
|
+
in which round and phase, and why, so "try again" has a request to retry.
|
|
691
|
+
- **Stop.** Pressing stop ends the turn as cancelled, and kills a running
|
|
692
|
+
command or check along with its whole process group.
|
|
693
|
+
|
|
694
|
+
### Flags and refusals
|
|
695
|
+
|
|
696
|
+
`volley acp` takes the run command's seat flags, and requires four of them:
|
|
697
|
+
`--builder-provider`, `--builder-model`, `--critic-provider`, and
|
|
698
|
+
`--critic-model`. A missing one is refused with every missing flag named. The
|
|
699
|
+
rest are optional: `--builder-url`, `--critic-url`, `--builder-max-steps`,
|
|
700
|
+
`--builder-max-tool-calls`, `--allow-unsandboxed-builder`, `--critic`, `--check`, `--max-iterations`,
|
|
701
|
+
`--max-cost-usd`, and `--state-dir`.
|
|
702
|
+
|
|
703
|
+
It refuses these before serving, each with exit 5:
|
|
704
|
+
|
|
705
|
+
| Refused | Why |
|
|
706
|
+
|---|---|
|
|
707
|
+
| `--worktree` | A sibling worktree sits outside the project the editor has open. |
|
|
708
|
+
| `--git` | It commits to the open repo after every phase, which no permission prompt covers. |
|
|
709
|
+
| `--json` | Stdout carries the protocol alone. |
|
|
710
|
+
| a `claude_cli` builder | The Claude CLI runs its own tools under its own permission model, so its edits would never reach the editor's prompts. |
|
|
711
|
+
|
|
712
|
+
A `claude_cli` critic is allowed: it only reads, and it runs on the host, where
|
|
713
|
+
its login works. The whole config resolves once at startup, so a bad flag or an
|
|
714
|
+
unset `OPENROUTER_API_KEY` exits 5 there, where Zed's agent log shows it, rather
|
|
715
|
+
than failing every prompt.
|
|
716
|
+
|
|
717
|
+
### Modes and approvals
|
|
718
|
+
|
|
719
|
+
Each session starts in Manual. Zed's mode picker switches it, and a switch holds
|
|
720
|
+
from the next call on, mid-turn included. Zed also saves the mode you pick as the
|
|
721
|
+
entry's `default_config_options`, and starts the entry's later threads in it.
|
|
722
|
+
|
|
723
|
+
| Mode | Asks before an edit | Asks before a command |
|
|
724
|
+
|---|---|---|
|
|
725
|
+
| Manual | yes | yes |
|
|
726
|
+
| Accept edits | no | yes |
|
|
727
|
+
| Auto | no | no; offered only contained or opted out (below) |
|
|
728
|
+
|
|
729
|
+
An edit asks with its diff, and offers "Allow all edits this session", which
|
|
730
|
+
switches the session to Accept edits. A command offers "Always allow" for that
|
|
731
|
+
exact command line: allowing `pnpm test` doesn't cover `pnpm test --watch=false`,
|
|
732
|
+
which asks again. It never offers one for a command with shell control syntax
|
|
733
|
+
such as `;` or `|`. A call you reject reaches the builder as
|
|
734
|
+
`tool_approval_denied`, and its prompt tells it not to reach the same result
|
|
735
|
+
another way.
|
|
736
|
+
|
|
737
|
+
- **Uncontained, Auto isn't offered.** Outside volley's
|
|
738
|
+
[sandbox container](#local-builder) (`VOLLEY_CONTAINED`), a person approving
|
|
739
|
+
each command takes the container's place, so the mode picker lists Manual and
|
|
740
|
+
Accept edits alone, and the builder's `run_command` asks in both. No client
|
|
741
|
+
can switch the session to Auto, mid-turn included. A Zed entry left on Auto
|
|
742
|
+
(`"default_config_options": { "mode": "auto" }`, from a time it was offered)
|
|
743
|
+
falls back to Manual, with a warning in Zed's log.
|
|
744
|
+
`--allow-unsandboxed-builder` or `VOLLEY_ALLOW_UNSANDBOXED_BUILDER=1` opts
|
|
745
|
+
out, and Auto is offered again. So unlike the run command, `volley acp` never
|
|
746
|
+
refuses an uncontained builder.
|
|
747
|
+
- **The check asks whenever a command would.** In Accept edits and Auto the
|
|
748
|
+
builder can rewrite a test without asking, and the check runs it next. So the
|
|
749
|
+
check asks on its own card exactly when a builder command would ask, offering
|
|
750
|
+
only Allow and Reject, since an "always" for checkride's `pnpm exec` would
|
|
751
|
+
cover the builder's commands too. A rejected check counts as a failed one, so
|
|
752
|
+
a round can't converge on work nobody checked.
|
|
753
|
+
- **No keys in commands.** The builder's commands and the check run with the
|
|
754
|
+
agent's environment minus `OPENROUTER_API_KEY` and `ANTHROPIC_API_KEY`,
|
|
755
|
+
`PATH` kept, so nothing the builder writes can read the key a seat runs on.
|
|
756
|
+
|
|
757
|
+
### Where its state goes
|
|
758
|
+
|
|
759
|
+
Nothing of volley's own lands in the repo. Each prompt's run state goes under
|
|
760
|
+
the same state root as any run's ([Run artifacts](#run-artifacts)):
|
|
761
|
+
`<root>/<run id>/`, where the root is `--state-dir`, else
|
|
762
|
+
`$XDG_STATE_HOME/volley`, else `~/.local/state/volley`. An unwritable root
|
|
763
|
+
stops `volley acp` at startup, so it shows once in the agent log. The whole
|
|
764
|
+
root is safe to delete whenever no prompt is running. A run's
|
|
765
|
+
`trajectory.jsonl` there replays in `fascicle-viewer` like a CLI run's.
|
|
766
|
+
|
|
767
|
+
### Zed setup
|
|
768
|
+
|
|
769
|
+
Run `pnpm build` first: Zed runs `dist/cli.js`, and nothing else rebuilds it.
|
|
770
|
+
Then add volley as a custom agent server in Zed's `settings.json`:
|
|
771
|
+
|
|
772
|
+
```json
|
|
773
|
+
{
|
|
774
|
+
"agent_servers": {
|
|
775
|
+
"Volley": {
|
|
776
|
+
"type": "custom",
|
|
777
|
+
"command": "node",
|
|
778
|
+
"args": [
|
|
779
|
+
"/abs/path/to/volley/dist/cli.js", "acp",
|
|
780
|
+
"--builder-provider", "openrouter", "--builder-model", "anthropic/claude-sonnet-5.5",
|
|
781
|
+
"--critic-provider", "openrouter", "--critic-model", "anthropic/claude-haiku-5.5",
|
|
782
|
+
"--check", "pnpm test"
|
|
783
|
+
],
|
|
784
|
+
"env": {
|
|
785
|
+
"OPENROUTER_API_KEY": "sk-or-v1-...",
|
|
786
|
+
"PATH": "/usr/local/bin:/usr/bin:/bin"
|
|
787
|
+
}
|
|
788
|
+
}
|
|
789
|
+
}
|
|
790
|
+
}
|
|
791
|
+
```
|
|
792
|
+
|
|
793
|
+
`PATH` is there for the builder's commands and the check. Each runs with
|
|
794
|
+
`/bin/sh -c`, which reads no shell profile, so a `node` or `pnpm` from nvm, fnm,
|
|
795
|
+
volta, or asdf can come back as `command not found` unless it's on this `PATH`.
|
|
796
|
+
Put the directories that `dirname "$(command -v node)"` and
|
|
797
|
+
`dirname "$(command -v pnpm)"` print at its front.
|
|
798
|
+
|
|
799
|
+
### Check it in Zed
|
|
800
|
+
|
|
801
|
+
Run this by hand against a scratch repository with a test command (a
|
|
802
|
+
`package.json` whose `test` script runs `node --test` is enough), using the
|
|
803
|
+
snippet above. Start a thread with "Volley", and check that the mode picker
|
|
804
|
+
shows Manual: Zed starts a thread in the last mode you picked, so an earlier run
|
|
805
|
+
of step 8 leaves it on Accept edits. Stay in Manual unless a step says
|
|
806
|
+
otherwise:
|
|
807
|
+
|
|
808
|
+
1. Ask for a small change. The builder's work should stream in as the reply,
|
|
809
|
+
under a **Round 1** line.
|
|
810
|
+
2. When it edits a file, a permission request should show the diff. Allow it,
|
|
811
|
+
and the card should keep the diff.
|
|
812
|
+
3. When it runs a command, a permission request should show the command before
|
|
813
|
+
it runs, and once allowed, the card should show the output.
|
|
814
|
+
4. Once the builder finishes, the check should ask, then run as a command card
|
|
815
|
+
titled `pnpm test`, with its output.
|
|
816
|
+
5. The critic's verdict should show as a thought.
|
|
817
|
+
6. The turn should end with the summary: whether it converged, in how many
|
|
818
|
+
rounds, and the last verdict.
|
|
819
|
+
7. Ask it to run `sleep 60`, allow it, and press stop. The turn should end as
|
|
820
|
+
cancelled, and `pgrep -fl 'sleep 60'` should find nothing.
|
|
821
|
+
8. Open the mode picker. It should list Manual and Accept edits, and no Auto,
|
|
822
|
+
since volley runs uncontained. Switch to Accept edits and ask for another
|
|
823
|
+
change. The edits shouldn't ask, but a command still should.
|
|
824
|
+
|
|
825
|
+
Two more round it out. Ask it to run `echo "key=$OPENROUTER_API_KEY"`, and the
|
|
826
|
+
output should be `key=` with nothing after it. And the repository should hold no
|
|
827
|
+
`.volley`.
|
|
828
|
+
|
|
497
829
|
## What the builder changed
|
|
498
830
|
|
|
499
831
|
A check that exits 0 proves the check passed. It does not prove the builder did
|
|
@@ -508,7 +840,7 @@ iteration and reports what moved:
|
|
|
508
840
|
which ones moved.
|
|
509
841
|
- Changed paths matching the **gate** — the tests, fixtures, and check
|
|
510
842
|
configuration that decide whether the work passes — are called out separately
|
|
511
|
-
in the prompt, listed in
|
|
843
|
+
in the prompt, listed in the run's `summary.json` as `comparison.gate_edits`,
|
|
512
844
|
and warned about on stderr as they happen.
|
|
513
845
|
|
|
514
846
|
A gate edit is not misconduct: plenty of tasks are *about* the tests. It is the
|
|
@@ -537,11 +869,15 @@ that `--fail-on-gate-edit` cannot fire there.
|
|
|
537
869
|
|
|
538
870
|
## Run artifacts
|
|
539
871
|
|
|
540
|
-
Everything volley knows lives in
|
|
872
|
+
Everything volley knows about a run lives in a directory of its own, outside
|
|
873
|
+
the workspace: `<root>/<run id>/`, where the root is `--state-dir` (or
|
|
874
|
+
`state_dir` in a config file), else `$XDG_STATE_HOME/volley`, else
|
|
875
|
+
`~/.local/state/volley`.
|
|
541
876
|
|
|
542
877
|
```
|
|
543
|
-
|
|
878
|
+
<root>/<run id>/
|
|
544
879
|
config.json # resolved run config (immutable per run)
|
|
880
|
+
config.sig # this machine's signature over config.json (none from inside the sandbox)
|
|
545
881
|
summary.json # run-level usage, cost, status
|
|
546
882
|
trajectory.jsonl # every fascicle trajectory event (replayable, below)
|
|
547
883
|
feedback.md # current critic feedback (harness-written)
|
|
@@ -549,20 +885,44 @@ Everything volley knows lives in the workspace:
|
|
|
549
885
|
iterations/NNN/ # per-iteration archive: feedback, verdict, check artifacts, summary
|
|
550
886
|
```
|
|
551
887
|
|
|
552
|
-
|
|
888
|
+
- **Why outside.** Some checks read every file in the tree, dot-directories
|
|
889
|
+
included, as markdownlint-cli2 and checkride's links walk do. Kept in the
|
|
890
|
+
workspace, the run's own `feedback.md` would fail them from the second
|
|
891
|
+
iteration on, a failure the builder can't fix. Kept out, the workspace holds
|
|
892
|
+
only the builder's work, and `--git` checkpoints never commit volley's files.
|
|
893
|
+
- **Finding a run.** A run prints `state: <dir>` when it starts and again in its
|
|
894
|
+
final summary, and `--json` and `summary.json` carry it as `run_state_dir`.
|
|
895
|
+
`--dry-run` names the root, and `--run-id` names the directory.
|
|
896
|
+
- **Unwritable.** A run directory volley can't write is a configuration error
|
|
897
|
+
(exit 5) before any spend. In the sandbox container, mount the run's own
|
|
898
|
+
directory ([Local builder](#local-builder)).
|
|
899
|
+
- **Pruning.** Nothing prunes old runs. A root volley creates is yours alone
|
|
900
|
+
(0700), since trajectories hold prompts and tool output. A reused `--run-id`
|
|
901
|
+
rotates the earlier run aside to `<run id>.bak.<timestamp>/`.
|
|
902
|
+
- **Older runs.** An older volley kept the same files in the workspace's
|
|
903
|
+
`.volley/`. `volley resume` still finds such a run there, and resumes it in
|
|
904
|
+
place once you pass `--trust-config`, since an older volley signed nothing
|
|
905
|
+
([Resume](#resume)).
|
|
906
|
+
|
|
907
|
+
`volley matrix` writes its sweep to `.volley-matrix/` in the workspace, and a
|
|
908
|
+
base config's `state_dir` moves each seat's run state as it would any run's.
|
|
553
909
|
|
|
554
910
|
The trajectory replays in fascicle's bundled viewer — the `fascicle-viewer` bin
|
|
555
911
|
ships inside the `fascicle` package; there is no separate viewer package:
|
|
556
912
|
|
|
557
913
|
```sh
|
|
558
|
-
pnpm dlx --package=fascicle fascicle-viewer
|
|
914
|
+
pnpm dlx --package=fascicle fascicle-viewer ~/.local/state/volley/<run id>/trajectory.jsonl
|
|
559
915
|
```
|
|
560
916
|
|
|
561
917
|
## Environment variables
|
|
562
918
|
|
|
563
919
|
See [.env.example](./.env.example): `VOLLEY_CLAUDE_BIN`, `VOLLEY_AUTH_MODE`,
|
|
564
|
-
`VOLLEY_LOG_LEVEL`, `VOLLEY_NO_COLOR`, `VOLLEY_PRICING_PATH`,
|
|
565
|
-
`ANTHROPIC_API_KEY` billing-meter warning
|
|
920
|
+
`VOLLEY_LOG_LEVEL`, `VOLLEY_NO_COLOR`, `VOLLEY_PRICING_PATH`, the
|
|
921
|
+
`ANTHROPIC_API_KEY` billing-meter warning, each seat's base URL
|
|
922
|
+
(`VOLLEY_OLLAMA_URL`, `VOLLEY_LMSTUDIO_URL`, `VOLLEY_OPENROUTER_URL`), and
|
|
923
|
+
`OPENROUTER_API_KEY` for [OpenRouter](#openrouter) seats. Every run reads
|
|
924
|
+
`XDG_STATE_HOME` for its default state root, and `XDG_CONFIG_HOME` for where
|
|
925
|
+
it keeps the key it signs run configs with.
|
|
566
926
|
|
|
567
927
|
## Development
|
|
568
928
|
|