@robmclarty/volley 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +123 -0
- package/LICENSE +201 -0
- package/README.md +514 -0
- package/dist/cli.js +3405 -0
- package/dist/cli.js.map +1 -0
- package/dist/index.d.ts +349 -0
- package/dist/index.js +2659 -0
- package/dist/index.js.map +1 -0
- package/package.json +94 -0
- package/src/builder/presets/harness_append_local.md +40 -0
- package/src/critic/presets/harness_append.md +15 -0
- package/src/critic/presets/harness_append_local.md +20 -0
- package/src/critic/presets/optimizer.md +8 -0
- package/src/critic/presets/researcher.md +7 -0
- package/src/critic/presets/reviewer.md +9 -0
package/README.md
ADDED
|
@@ -0,0 +1,514 @@
|
|
|
1
|
+
# volley
|
|
2
|
+
|
|
3
|
+
[](https://github.com/robmclarty/volley/actions/workflows/ci.yml)
|
|
4
|
+
|
|
5
|
+
A Converging Loop:
|
|
6
|
+
A CLI harness that runs a **builder/critic loop** until a task passes both a
|
|
7
|
+
deterministic check pipeline and a natural-language acceptance review.
|
|
8
|
+
|
|
9
|
+
volley is built on two substrates:
|
|
10
|
+
|
|
11
|
+
- [fascicle](https://www.npmjs.com/package/fascicle) — the loop primitive, the
|
|
12
|
+
`claude_cli` provider (one call = one complete agentic Claude Code session),
|
|
13
|
+
usage/cost accounting, trajectory logging, and cancellation.
|
|
14
|
+
- [checkride](https://www.npmjs.com/package/checkride) — the deterministic
|
|
15
|
+
check: one command, exit 0 = done, with a stable `.check/summary.json`
|
|
16
|
+
contract whose raw per-tool output is fed straight to the critic.
|
|
17
|
+
|
|
18
|
+
## Status
|
|
19
|
+
|
|
20
|
+
volley is a personal research harness. I built it to study builder/critic
|
|
21
|
+
loops on local models, and that is what I use it for. It is not (yet) meant
|
|
22
|
+
for broad use:
|
|
23
|
+
|
|
24
|
+
- **v0.x.** CLI flags, the config shape, and the `--json` output change
|
|
25
|
+
without notice. Read [CHANGELOG.md](./CHANGELOG.md) before upgrading.
|
|
26
|
+
- **Pinned substrates.** It pins specific versions of fascicle and checkride
|
|
27
|
+
and is only tested against those.
|
|
28
|
+
- **A local builder runs a real shell.** Read [Local builder](#local-builder)
|
|
29
|
+
before pointing one at anything you care about.
|
|
30
|
+
- **No support commitment.** Issues and findings are welcome; open an issue
|
|
31
|
+
before sending a large PR.
|
|
32
|
+
|
|
33
|
+
The design record lives in [`research/`](./research/) (specs and run
|
|
34
|
+
findings) and [`.plumbbob/`](./.plumbbob/) (build plans and logs).
|
|
35
|
+
|
|
36
|
+
## Set a cost cap
|
|
37
|
+
|
|
38
|
+
**Start here.** A volley run is a loop of complete agentic sessions, so it
|
|
39
|
+
spends faster than anything you drive by hand. With `ANTHROPIC_API_KEY` set,
|
|
40
|
+
every session bills the key at API rates. On a subscription, `claude_cli`
|
|
41
|
+
draws from your plan's usage limits — Anthropic announced a separate metered
|
|
42
|
+
credit for programmatic use in May 2026, then
|
|
43
|
+
[paused it on June 15](https://support.claude.com/en/articles/15036540-use-the-claude-agent-sdk-with-your-claude-plan)
|
|
44
|
+
before it took effect. Either way, always run with `--max-cost-usd`:
|
|
45
|
+
|
|
46
|
+
```sh
|
|
47
|
+
volley \
|
|
48
|
+
--prompt "Implement the TODO items in src/parser.ts" \
|
|
49
|
+
--workspace ./my-project \
|
|
50
|
+
--criteria "All TODOs resolved; existing tests still pass" \
|
|
51
|
+
--max-cost-usd 10
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
The cap is enforced in the loop guard after each phase. Sessions are never
|
|
55
|
+
killed mid-flight for cost (a builder killed mid-edit leaves the workspace
|
|
56
|
+
undefined), so set the cap below your true ceiling by roughly one iteration's
|
|
57
|
+
expected cost. Per-run totals land in `.volley/summary.json` and the final
|
|
58
|
+
terminal summary.
|
|
59
|
+
|
|
60
|
+
## How it works
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
prompt ──> builder (full agentic session in your workspace)
|
|
64
|
+
│
|
|
65
|
+
check (checkride --json, custom command, or none)
|
|
66
|
+
│
|
|
67
|
+
critic (read-only session; schema-validated verdict)
|
|
68
|
+
│
|
|
69
|
+
approved + check green? ── no ──> feedback → next iteration
|
|
70
|
+
│ yes
|
|
71
|
+
done
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
- **Builder** gets the task, the acceptance criteria, and the previous
|
|
75
|
+
iteration's critic feedback. It has full tools (Read/Write/Edit/Bash) and
|
|
76
|
+
runs with `--permission-mode acceptEdits` by default.
|
|
77
|
+
- **Check** is auto-detected: if the workspace uses checkride, volley runs
|
|
78
|
+
`pnpm exec checkride --json` and gates on the summary. Any shell command
|
|
79
|
+
works too (`--check "npm test"`), or `--check none` for critic-only runs.
|
|
80
|
+
- **Critic** is read-only (`Read`, `Grep`, `Glob` — no write path at all) and
|
|
81
|
+
returns a schema-validated `{ verdict, feedback, unmet_criteria }`. It is told
|
|
82
|
+
which files the builder changed, so it reviews a change rather than a tree.
|
|
83
|
+
The harness writes `.volley/feedback.md` and `.volley/verdict`; the critic
|
|
84
|
+
cannot get them wrong.
|
|
85
|
+
|
|
86
|
+
Each iteration is a fresh session — no context carries over except through the
|
|
87
|
+
workspace filesystem and the critic's feedback. Swap the critic to change what
|
|
88
|
+
the loop optimizes for: `--critic reviewer` (default), `optimizer`,
|
|
89
|
+
`researcher`, or a path to your own system prompt.
|
|
90
|
+
|
|
91
|
+
## Install
|
|
92
|
+
|
|
93
|
+
Requires Node ≥ 24, pnpm, and the [Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code)
|
|
94
|
+
installed and authenticated.
|
|
95
|
+
|
|
96
|
+
From a checkout:
|
|
97
|
+
|
|
98
|
+
```sh
|
|
99
|
+
git clone https://github.com/robmclarty/volley.git && cd volley
|
|
100
|
+
pnpm install && pnpm build
|
|
101
|
+
pnpm link --global # puts `volley` on your PATH
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Or from npm, where the package is scoped (the bare `volley` name belongs to an
|
|
105
|
+
unrelated project); the binary is still `volley`:
|
|
106
|
+
|
|
107
|
+
```sh
|
|
108
|
+
pnpm add -g @robmclarty/volley
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## Usage
|
|
112
|
+
|
|
113
|
+
```
|
|
114
|
+
volley [options]
|
|
115
|
+
volley --config <path>
|
|
116
|
+
volley resume <run-id>
|
|
117
|
+
volley matrix --builders <models> --critics <models> --config <path>
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
| Flag | Default | Description |
|
|
121
|
+
|---|---|---|
|
|
122
|
+
| `--prompt` | (required) | Task prompt. Literal string or `@path/to/file.md`. |
|
|
123
|
+
| `--workspace` | (required) | Workspace directory. Must exist and be writable. |
|
|
124
|
+
| `--criteria` | (required) | Acceptance criteria. Literal string or `@path`. |
|
|
125
|
+
| `--check` | `auto` | `auto` (checkride if detected, else none), `none`, or a shell command. |
|
|
126
|
+
| `--builder-model` | `opus` | `opus`/`sonnet`/`haiku` or a full model id. |
|
|
127
|
+
| `--builder-provider` | `claude_cli` | `claude_cli`, `ollama`, or `lmstudio` (see [Local builder](#local-builder)). |
|
|
128
|
+
| `--builder-max-steps` | `50` | Local builder tool-loop step cap per iteration. Ignored for `claude_cli`. |
|
|
129
|
+
| `--allow-unsandboxed-builder` | off | Permit a local builder to run without a sandbox (see [Local builder](#local-builder)). |
|
|
130
|
+
| `--critic-model` | `opus` | Model for the critic. |
|
|
131
|
+
| `--critic-provider` | `claude_cli` | `claude_cli`, `ollama`, or `lmstudio` (see [Local critic](#local-critic)). |
|
|
132
|
+
| `--builder-permission-mode` | `acceptEdits` | Or `bypassPermissions` for fully trusted workspaces. |
|
|
133
|
+
| `--critic` | `reviewer` | `reviewer`, `optimizer`, `researcher`, or a prompt file path. |
|
|
134
|
+
| `--max-iterations` | `10` | Hard iteration cap. |
|
|
135
|
+
| `--max-cost-usd` | none | Hard USD ceiling. Set it. |
|
|
136
|
+
| `--git` | off | Commit after each phase (workspace must be a git repo). |
|
|
137
|
+
| `--worktree` | off | Isolate the builder run in a per-run git worktree (workspace must be a git repo). Without `--git`, a successful run's work is kept on the run branch — see [Local builder](#local-builder). |
|
|
138
|
+
| `--discard-worktree` | off | Throw a `--worktree` run's effects away at teardown instead of keeping them on the run branch. Requires `--worktree`. |
|
|
139
|
+
| `--gate-paths` | (built-in list) | Comma-separated globs naming the gate — tests, fixtures, check config. Replaces the defaults; see [What the builder changed](#what-the-builder-changed). |
|
|
140
|
+
| `--fail-on-gate-edit` | off | Halt the run (exit 8) if the builder edits a gate path, instead of reporting it. |
|
|
141
|
+
| `--sandbox-image` | `volley-sandbox:latest` | Container image for the local-builder sandbox. Or `VOLLEY_SANDBOX_IMAGE`. |
|
|
142
|
+
| `--dry-run` | off | Validate config, run `checkride doctor` and (for a local critic) the critic-seat canary, then exit. |
|
|
143
|
+
| `--config` | — | TypeScript config file (`VolleyConfig` default export). CLI flags override. |
|
|
144
|
+
| `--json` | off | Machine mode: one summary JSON document on stdout. |
|
|
145
|
+
| `--verbose` | off | Full tool inputs/outputs (truncated at 4000 chars). |
|
|
146
|
+
| `--quiet` | off | Phase transitions, cost lines, and the final summary only. |
|
|
147
|
+
| `--no-thinking` | off | Hide reasoning chunks (still recorded in the trajectory). |
|
|
148
|
+
|
|
149
|
+
Human-readable progress goes to **stderr**; stdout carries machine output
|
|
150
|
+
only, so volley composes under bigger harnesses the same way it composes
|
|
151
|
+
checkride.
|
|
152
|
+
|
|
153
|
+
### Exit codes
|
|
154
|
+
|
|
155
|
+
| Code | Meaning |
|
|
156
|
+
|---|---|
|
|
157
|
+
| 0 | Check passed and verdict approved. |
|
|
158
|
+
| 2 | Max iterations reached without success. |
|
|
159
|
+
| 3 | Builder/provider error. |
|
|
160
|
+
| 4 | Check pipeline error (broken pipeline, not failing checks). |
|
|
161
|
+
| 5 | Configuration error. |
|
|
162
|
+
| 6 | Critic error (schema validation failed after repair attempts). |
|
|
163
|
+
| 7 | Cost cap reached. |
|
|
164
|
+
| 8 | The builder edited the gate and `--fail-on-gate-edit` was set. |
|
|
165
|
+
| 130 | Interrupted (SIGINT); resumable. |
|
|
166
|
+
|
|
167
|
+
### Config file
|
|
168
|
+
|
|
169
|
+
```ts
|
|
170
|
+
// volley.config.ts
|
|
171
|
+
import type { VolleyConfig } from '@robmclarty/volley';
|
|
172
|
+
|
|
173
|
+
const config: VolleyConfig = {
|
|
174
|
+
prompt: '...',
|
|
175
|
+
workspace: './my-project',
|
|
176
|
+
criteria: '...',
|
|
177
|
+
check: 'auto',
|
|
178
|
+
builder_model: 'opus',
|
|
179
|
+
critic_model: 'sonnet',
|
|
180
|
+
critic: 'reviewer',
|
|
181
|
+
max_iterations: 10,
|
|
182
|
+
max_cost_usd: 20.0,
|
|
183
|
+
};
|
|
184
|
+
|
|
185
|
+
export default config;
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
### Resume
|
|
189
|
+
|
|
190
|
+
```sh
|
|
191
|
+
volley resume <run-id> # in the workspace directory
|
|
192
|
+
volley resume <run-id> --workspace ./my-project
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
Continues from the last completed iteration with the saved feedback and cost
|
|
196
|
+
totals. An iteration interrupted mid-phase is discarded and re-run.
|
|
197
|
+
|
|
198
|
+
### Matrix sweep
|
|
199
|
+
|
|
200
|
+
Finding the best builder×critic pairing means running the same task against
|
|
201
|
+
several model combos and comparing. `volley matrix` sweeps the cross product
|
|
202
|
+
serially over one fixed config instead of hand-editing model flags per run:
|
|
203
|
+
|
|
204
|
+
```sh
|
|
205
|
+
volley matrix \
|
|
206
|
+
--config ./examples/essayist/volley.config.ts \
|
|
207
|
+
--builders qwen3.6:latest \
|
|
208
|
+
--critics qwen3:8b,gemma4:12b,glm-4.7-flash \
|
|
209
|
+
--repeat 3
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
| Matrix flag | Description |
|
|
213
|
+
|---|---|
|
|
214
|
+
| `--config` | Base `VolleyConfig` (TS/JS): the task, workspace, providers, and caps held fixed. Required. |
|
|
215
|
+
| `--builders` | Comma-separated builder models to sweep. |
|
|
216
|
+
| `--critics` | Comma-separated critic models to sweep. |
|
|
217
|
+
| `--repeat` | Attempts per seat (default 1). A pass *rate* needs more than one. |
|
|
218
|
+
| `--workspace` | Override the base config's workspace. |
|
|
219
|
+
| `--json` | Machine mode: the aggregate as JSON on stdout, no table. |
|
|
220
|
+
| `--verbose` | Show full builder/critic streams per combo. |
|
|
221
|
+
| `--quiet` | Per-combo phase transitions and the final table only. |
|
|
222
|
+
|
|
223
|
+
`--builders` and `--critics` are comma-separated model lists; every combination
|
|
224
|
+
runs `--repeat` times (`--builder-model` × `--critic-model`), in series — the
|
|
225
|
+
local providers share one GPU, so parallel combos would thrash the model loader.
|
|
226
|
+
Each run forces `--worktree --discard-worktree` for a clean per-run reset — a
|
|
227
|
+
sweep wants verdicts, so each seat's effects are isolated and then thrown away,
|
|
228
|
+
leaving neither a branch nor a commit behind — so the **workspace must be a git
|
|
229
|
+
repository**. Every attempt's run state is written to
|
|
230
|
+
`.volley-matrix/<builder>__<critic>/run-NN/summary.json`, the whole sweep to
|
|
231
|
+
`.volley-matrix/matrix.json`, and one aggregate table goes to stderr:
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
builder critic pass iters wall cost salvage flags why
|
|
235
|
+
────────────── ───────────── ──── ───── ─────── ───── ─────── ───── ──────────────────
|
|
236
|
+
qwen3.6:latest qwen3:8b 3/3 1.0 403.4s $0.00 0% — —
|
|
237
|
+
qwen3.6:latest gemma4:12b 2/3 1.5 512.0s $0.00 0% — unmet: 2 criteria
|
|
238
|
+
qwen3.6:latest glm-4.7-flash 0/3 — 184.0s $0.00 12% deg check: types, test
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
**Why `--repeat`.** One run answers "did this pairing converge that time", and
|
|
242
|
+
that is all n=1 can honestly answer: local runs are stochastic. A single sample
|
|
243
|
+
supports a *hard* failure (the qwen3.6 critic death reproduced 2/2 in
|
|
244
|
+
[`research/v3-comparison-finding.md`](./research/v3-comparison-finding.md)) and
|
|
245
|
+
nothing else — not iterations-to-converge, not wall clock. With `--repeat` the
|
|
246
|
+
`pass` column is a rate, and the averages behind it are over the attempts that
|
|
247
|
+
can support them: `iters` averages the *converged* attempts only, `wall` and
|
|
248
|
+
`cost` average every attempt that reported one, and `salvage` is a ratio of
|
|
249
|
+
totals rather than a mean of ratios.
|
|
250
|
+
|
|
251
|
+
**Why a seat fell off** is the `why` column — the point of a ladder sweep.
|
|
252
|
+
`budget_exhausted` is a status, not a diagnosis, so the row instead names the
|
|
253
|
+
failing check slots, the criteria the critic still judged unmet, a cost cap, a
|
|
254
|
+
gate edit, or the error that broke the run. `flags` carries what qualifies a
|
|
255
|
+
pass rather than explaining a failure: `deg` for a degraded critic verdict,
|
|
256
|
+
`gate` for a builder that edited the gate. Every attempt's full detail —
|
|
257
|
+
statuses, verdicts, failing slots, unmet criteria, gate edits — is in the
|
|
258
|
+
`--json` aggregate and in `matrix.json`.
|
|
259
|
+
|
|
260
|
+
A run that does not converge is a *result*, shown in the table; the sweep exits
|
|
261
|
+
0 as long as every attempt produced a summary, and nonzero only when one
|
|
262
|
+
produced none at all (`status: broke`).
|
|
263
|
+
|
|
264
|
+
## Local critic
|
|
265
|
+
|
|
266
|
+
The builder always runs on the Claude Code CLI, but the critic is read-only
|
|
267
|
+
and schema-constrained — a good fit for a cheaper local model. Point it at a
|
|
268
|
+
local provider with `--critic-provider`:
|
|
269
|
+
|
|
270
|
+
```sh
|
|
271
|
+
volley \
|
|
272
|
+
--prompt "@task.md" --workspace ./my-project --criteria "@criteria.md" \
|
|
273
|
+
--critic-provider ollama --critic-model qwen3:32b \
|
|
274
|
+
--max-cost-usd 10
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
Instead of the CLI's built-in Read/Grep/Glob, a local critic gets volley's own
|
|
278
|
+
workspace-scoped read-only tools — `read_file`, `search_files`, `list_files` —
|
|
279
|
+
each confined to the workspace root (no write path, no path traversal). The
|
|
280
|
+
structured verdict contract is identical; fascicle validates and repairs the
|
|
281
|
+
local model's output the same way.
|
|
282
|
+
|
|
283
|
+
Providers and their setup:
|
|
284
|
+
|
|
285
|
+
| `--critic-provider` | Server | Base-URL env (default) | Required peer dependency |
|
|
286
|
+
|---|---|---|---|
|
|
287
|
+
| `ollama` | [Ollama](https://ollama.com) | `VOLLEY_OLLAMA_URL` (`http://localhost:11434`) | `ai-sdk-ollama@^4` |
|
|
288
|
+
| `lmstudio` | [LM Studio](https://lmstudio.ai) | `VOLLEY_LMSTUDIO_URL` (`http://localhost:1234/v1`) | `@ai-sdk/openai-compatible` |
|
|
289
|
+
|
|
290
|
+
Install the peer for your provider in the project running volley (e.g.
|
|
291
|
+
`pnpm add ai-sdk-ollama@^4`); fascicle loads it lazily only when the local
|
|
292
|
+
critic actually runs. The major matters: it must satisfy fascicle's declared
|
|
293
|
+
peer range (`^4` for fascicle 0.9.5, which tracks the AI SDK v7 line; a bare
|
|
294
|
+
`pnpm add ai-sdk-ollama` installs v4, which is exactly what this major wants).
|
|
295
|
+
The Ollama base URL is the **server root** — `ai-sdk-ollama` adds the `/api`
|
|
296
|
+
prefix itself (volley strips a trailing `/api` from `VOLLEY_OLLAMA_URL` for
|
|
297
|
+
compatibility). Local providers are free, so the critic's cost is reported as
|
|
298
|
+
`$0.000` and the run total reflects builder spend only.
|
|
299
|
+
|
|
300
|
+
Keeping the builder on Claude while the critic goes local is the low-risk split:
|
|
301
|
+
the recurring per-iteration critic cost drops to zero and the critic is
|
|
302
|
+
read-only, so a weaker local model can only mis-judge, never mis-edit. A local
|
|
303
|
+
*builder* is also supported ([below](#local-builder)) but asks more of you — it
|
|
304
|
+
gets a real host bash, so it is refused until you opt in.
|
|
305
|
+
|
|
306
|
+
**A caveat on the critic model.** The critic seat is the one place volley pairs a
|
|
307
|
+
tool surface with a constrained structured verdict, and not every local model
|
|
308
|
+
survives that combination. `qwen3.6:latest` in the critic seat reproducibly dies
|
|
309
|
+
on Ollama's server-side tool-call XML parser (`qwen35.go` / `qwen3coder.go`) —
|
|
310
|
+
the *same* model drives the builder tool loop fine, so the defect is specific to
|
|
311
|
+
the constrained-verdict path. volley no longer lets that kill a run: the critic
|
|
312
|
+
phase degrades (a bounded retry, then a tool-less fallback that keeps the
|
|
313
|
+
schema), marks the verdict `critic_degraded: true`, and `--dry-run` predicts the
|
|
314
|
+
doomed combo up front with a critic-seat canary. Even so, prefer a critic whose
|
|
315
|
+
tool-call encoding Ollama parses cleanly — `qwen3:8b`, `gemma4:12b`, and
|
|
316
|
+
`glm-4.7-flash` all pass. The full model-vs-transport write-up is in
|
|
317
|
+
[`research/v3-comparison-finding.md`](./research/v3-comparison-finding.md).
|
|
318
|
+
|
|
319
|
+
## Local builder
|
|
320
|
+
|
|
321
|
+
`--builder-provider ollama|lmstudio` runs the builder on a local model too. A
|
|
322
|
+
local model brings no built-in tools, so volley supplies the whole agentic
|
|
323
|
+
surface itself — `read_file`, `search_files`, `list_files`, `write_file`,
|
|
324
|
+
`edit_file`, `bash`, `fetch`, and an explicit `finish` — and runs one bounded
|
|
325
|
+
tool loop per iteration (capped by `--builder-max-steps`, default 50; hitting
|
|
326
|
+
the cap is treated as partial work, not an error). The produced workspace goes
|
|
327
|
+
to check + critic exactly like a `claude_cli` build.
|
|
328
|
+
|
|
329
|
+
```sh
|
|
330
|
+
volley \
|
|
331
|
+
--prompt "@task.md" --workspace ./my-project --criteria "@criteria.md" \
|
|
332
|
+
--builder-provider ollama --builder-model qwen3-coder:30b \
|
|
333
|
+
--allow-unsandboxed-builder \
|
|
334
|
+
--check "pnpm check"
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
**A local builder gets a real host `bash`, so it is refused unless contained.**
|
|
338
|
+
Unlike the read-only critic, the builder gets a real `bash` (write + exec) in
|
|
339
|
+
your workspace. So a local builder is refused *before any model spend* unless
|
|
340
|
+
volley detects it is running *inside* its own hardened sandbox container — the
|
|
341
|
+
operator launches volley there with `docker run <hardened flags>
|
|
342
|
+
volley-sandbox:latest volley …`, which sets `VOLLEY_CONTAINED=1` (baked into the
|
|
343
|
+
image; volley detects containment, it does not start the container itself). The
|
|
344
|
+
container gives default-deny network egress with a host-gateway allowlist; the
|
|
345
|
+
hardened `docker run` spec lives in `src/sandbox.ts`, and `--sandbox-image` /
|
|
346
|
+
`VOLLEY_SANDBOX_IMAGE` overrides the default `volley-sandbox:latest`. Inside the
|
|
347
|
+
container, `claude_cli`'s subscription/OAuth token does not survive — it is
|
|
348
|
+
mangled crossing the boundary — so a *contained* Claude role must drive by API
|
|
349
|
+
key (`VOLLEY_AUTH_MODE=api_key` with `ANTHROPIC_API_KEY`); the all-Claude
|
|
350
|
+
subscription path stays on the host, Docker-free.
|
|
351
|
+
|
|
352
|
+
To run a local builder uncontained on the host anyway, opt out with
|
|
353
|
+
`--allow-unsandboxed-builder` (or `VOLLEY_ALLOW_UNSANDBOXED_BUILDER=1`), which
|
|
354
|
+
prints a one-time warning — only point it at a workspace you are willing to let a
|
|
355
|
+
local model run shell commands in.
|
|
356
|
+
|
|
357
|
+
Independently of the sandbox, `--worktree` isolates a run in a per-run git
|
|
358
|
+
worktree on a branch named `volley/<run id>` (the workspace must be a git repo),
|
|
359
|
+
so the builder's effects never touch your working tree while the loop runs. What
|
|
360
|
+
becomes of those effects when the run converges depends on two flags — volley
|
|
361
|
+
prints which one applies at `--dry-run` and again at run start, so it is never a
|
|
362
|
+
surprise at teardown:
|
|
363
|
+
|
|
364
|
+
| Flags | A successful run's work |
|
|
365
|
+
|---|---|
|
|
366
|
+
| `--worktree --git` | Squash-merged onto the workspace branch, then the run branch is deleted. |
|
|
367
|
+
| `--worktree` | Committed onto `volley/<run id>` and left there. Recover it with `git switch volley/<run id>` or `git cherry-pick`; the branch is named in the final summary and in `--json` as `salvaged_branch`. |
|
|
368
|
+
| `--worktree --discard-worktree` | Thrown away with the branch — verdicts only, nothing kept. |
|
|
369
|
+
|
|
370
|
+
Any run that does *not* converge (cost cap, budget exhausted, interrupt, error)
|
|
371
|
+
is discarded wholesale in all three modes: an abandoned phase leaves nothing
|
|
372
|
+
behind.
|
|
373
|
+
|
|
374
|
+
Provider setup is the same as the [local critic](#local-critic) table
|
|
375
|
+
(`VOLLEY_OLLAMA_URL` / `VOLLEY_LMSTUDIO_URL`, same peer dependencies). At
|
|
376
|
+
builder start (and before a local critic's first verdict) volley pre-loads an
|
|
377
|
+
Ollama model with a load-only `/api/generate` call, so a cold multi-GB model
|
|
378
|
+
load doesn't eat the first real request's time-to-first-byte budget — without
|
|
379
|
+
it, a big cold model can die minutes in with an opaque
|
|
380
|
+
`stream interrupted: fetch failed`. (That cold-load timeout is an
|
|
381
|
+
ai-sdk/undici in-request limit, not an Ollama one: on fascicle's native
|
|
382
|
+
transport — a documented future option, not the path volley runs today — the
|
|
383
|
+
per-turn call is a raw `fetch` with no in-request timeout, so this pre-load's
|
|
384
|
+
necessity drops away and the `num_ctx`/`keep_alive` levers below move to
|
|
385
|
+
per-call `provider_options.ollama`.) Getting a
|
|
386
|
+
weak local model to drive a tool loop reliably has three sharp edges:
|
|
387
|
+
|
|
388
|
+
- **Set the context length to ≥ ~16k tokens.** Ollama's 4k default silently
|
|
389
|
+
truncates the tool schemas volley sends, which is the single most common cause
|
|
390
|
+
of a local model "ignoring" its tools — set `num_ctx` (e.g. a Modelfile
|
|
391
|
+
`PARAMETER num_ctx 16384`, or the request-level option) before you blame the
|
|
392
|
+
prompt. volley warns at builder start when it can read a too-small `num_ctx`
|
|
393
|
+
off the Ollama server, but a window left at the server default is not
|
|
394
|
+
reported back, so treat this as a required setup step rather than something
|
|
395
|
+
the harness is guaranteed to catch.
|
|
396
|
+
- **Pin the model, runtime, and parser as one unit.** Tool-call dialects differ
|
|
397
|
+
by model family (Qwen3 emits Hermes-style JSON; Qwen3-Coder emits XML), and
|
|
398
|
+
crossing a model with the wrong runtime parser silently drops tool calls.
|
|
399
|
+
volley salvages a call emitted as plain assistant text where it can (and
|
|
400
|
+
records a per-run salvage rate in the iteration summary), but that is
|
|
401
|
+
insurance, not a substitute for a matched runtime.
|
|
402
|
+
- **LM Studio 0.4.1+ has an Anthropic-compatible fallback.** If OpenAI-compat
|
|
403
|
+
tool-call parsing misbehaves for a Qwen-class model, LM Studio 0.4.1+ ships an
|
|
404
|
+
Anthropic-style `/v1/messages` endpoint (point `VOLLEY_LMSTUDIO_URL` at it)
|
|
405
|
+
that field reports find more reliable for tool calling.
|
|
406
|
+
|
|
407
|
+
Local providers are free, so a local builder's cost is reported as `$0.000` and
|
|
408
|
+
never trips `--max-cost-usd`.
|
|
409
|
+
|
|
410
|
+
## What the builder changed
|
|
411
|
+
|
|
412
|
+
A check that exits 0 proves the check passed. It does not prove the builder did
|
|
413
|
+
the work — a builder can also pass by editing the test. So every iteration,
|
|
414
|
+
volley diffs the build root against a baseline captured before the first
|
|
415
|
+
iteration and reports what moved:
|
|
416
|
+
|
|
417
|
+
- The **critic prompt** carries the changed-path list, so the critic reviews the
|
|
418
|
+
change instead of re-reading a tree the check already blessed. Paths and
|
|
419
|
+
statuses only, never diff hunks — the list stays bounded for a local critic
|
|
420
|
+
with a small context window, which can read any file it wants once it knows
|
|
421
|
+
which ones moved.
|
|
422
|
+
- Changed paths matching the **gate** — the tests, fixtures, and check
|
|
423
|
+
configuration that decide whether the work passes — are called out separately
|
|
424
|
+
in the prompt, listed in `.volley/summary.json` as `comparison.gate_edits`,
|
|
425
|
+
and warned about on stderr as they happen.
|
|
426
|
+
|
|
427
|
+
A gate edit is not misconduct: plenty of tasks are *about* the tests. It is the
|
|
428
|
+
fact that makes a green check prove less, so volley reports it by default rather
|
|
429
|
+
than refusing it. To refuse it, `--fail-on-gate-edit` halts the run on the spot
|
|
430
|
+
with exit 8 — success does not override it, because the pass is the thing the
|
|
431
|
+
edit puts in question.
|
|
432
|
+
|
|
433
|
+
The default gate is a broad, language-general list (`*.test.*`, `test/**`,
|
|
434
|
+
`fixtures/**`, `vitest.config.*`, `package.json`, `.github/workflows/**`, and
|
|
435
|
+
similar). `--gate-paths 'schema/**,docs/spec.md'` (or `gate_paths` in a config
|
|
436
|
+
file) **replaces** it — a task that owns its tests should say so — and an empty
|
|
437
|
+
list turns the report off entirely.
|
|
438
|
+
|
|
439
|
+
```sh
|
|
440
|
+
# The tests are the spec here: refuse a builder that edits them.
|
|
441
|
+
volley --prompt "@task.md" --workspace ./my-project --criteria "@criteria.md" \
|
|
442
|
+
--fail-on-gate-edit
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
Change detection needs git: it diffs against a baseline commit in the build root
|
|
446
|
+
(the worktree under `--worktree`, else the workspace). In a workspace that is not
|
|
447
|
+
a git repository there is no baseline, so the section is omitted from the critic
|
|
448
|
+
prompt rather than claiming the builder changed nothing — and `--dry-run` warns
|
|
449
|
+
that `--fail-on-gate-edit` cannot fire there.
|
|
450
|
+
|
|
451
|
+
## Run artifacts
|
|
452
|
+
|
|
453
|
+
Everything volley knows lives in the workspace:
|
|
454
|
+
|
|
455
|
+
```
|
|
456
|
+
.volley/
|
|
457
|
+
config.json # resolved run config (immutable per run)
|
|
458
|
+
summary.json # run-level usage, cost, status
|
|
459
|
+
trajectory.jsonl # every fascicle trajectory event (replayable, below)
|
|
460
|
+
feedback.md # current critic feedback (harness-written)
|
|
461
|
+
verdict # approved | changes_requested (harness-written)
|
|
462
|
+
iterations/NNN/ # per-iteration archive: feedback, verdict, check artifacts, summary
|
|
463
|
+
```
|
|
464
|
+
|
|
465
|
+
A stale `.volley/` from a previous run is rotated to `.volley.bak.<timestamp>/`.
|
|
466
|
+
|
|
467
|
+
The trajectory replays in fascicle's bundled viewer — the `fascicle-viewer` bin
|
|
468
|
+
ships inside the `fascicle` package; there is no separate viewer package:
|
|
469
|
+
|
|
470
|
+
```sh
|
|
471
|
+
pnpm dlx --package=fascicle fascicle-viewer .volley/trajectory.jsonl
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
## Environment variables
|
|
475
|
+
|
|
476
|
+
See [.env.example](./.env.example): `VOLLEY_CLAUDE_BIN`, `VOLLEY_AUTH_MODE`,
|
|
477
|
+
`VOLLEY_LOG_LEVEL`, `VOLLEY_NO_COLOR`, `VOLLEY_PRICING_PATH`, and the
|
|
478
|
+
`ANTHROPIC_API_KEY` billing-meter warning.
|
|
479
|
+
|
|
480
|
+
## Development
|
|
481
|
+
|
|
482
|
+
```sh
|
|
483
|
+
pnpm install
|
|
484
|
+
pnpm test # vitest
|
|
485
|
+
pnpm typecheck # tsc --noEmit
|
|
486
|
+
pnpm check # checkride — volley dogfoods its own definition of done
|
|
487
|
+
pnpm build # tsup → dist/
|
|
488
|
+
```
|
|
489
|
+
|
|
490
|
+
Live smoke tests against the real `claude` CLI and real checkride are opt-in:
|
|
491
|
+
|
|
492
|
+
```sh
|
|
493
|
+
VOLLEY_LIVE=1 pnpm test
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
The local-builder live test needs a running local runtime, so it gates on an
|
|
497
|
+
extra opt-in naming the provider that is actually up:
|
|
498
|
+
|
|
499
|
+
```sh
|
|
500
|
+
VOLLEY_LIVE=1 VOLLEY_LIVE_BUILDER_PROVIDER=ollama \
|
|
501
|
+
VOLLEY_LIVE_BUILDER_MODEL=qwen3-coder:30b pnpm test
|
|
502
|
+
```
|
|
503
|
+
|
|
504
|
+
## Relationship to ridgeline
|
|
505
|
+
|
|
506
|
+
volley is a deliberate sibling to
|
|
507
|
+
[ridgeline](https://github.com/robmclarty/ridgeline), not a replacement. ridgeline
|
|
508
|
+
fixes plan/build/evaluate as three phases with strong context boundaries;
|
|
509
|
+
volley collapses planning into the builder's autonomy and makes evaluation
|
|
510
|
+
fully pluggable. Same substrate (fascicle), different opinions.
|
|
511
|
+
|
|
512
|
+
## License
|
|
513
|
+
|
|
514
|
+
[Apache 2.0](./LICENSE)
|