@withgauge/cli 0.9.0 → 0.9.1-internal.20261019.00b83ea

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -6,6 +6,7 @@ live from your terminal.
6
6
 
7
7
  ```sh
8
8
  npm install -g @withgauge/cli
9
+ gauge update # update the CLI to the latest published version
9
10
  gauge onboard # sign in, create a workspace, seed starter measurements
10
11
  gauge auth login # (or just sign in)
11
12
  gauge orgs use <org> # set your default organization
@@ -13,25 +14,322 @@ gauge orgs use <org> # set your default organization
13
14
 
14
15
  Requires Node.js ≥ 22.12.
15
16
 
17
+ To install the optional
18
+ [`gauge-agents` skill](https://github.com/gauge-sh/gauge-skills/tree/main/gauge-agents)
19
+ globally, run:
20
+
21
+ ```sh
22
+ npx --yes skills@1.5.0 add gauge-sh/gauge-skills --skill gauge-agents --global --yes
23
+ ```
24
+
16
25
  ## Usage
17
26
 
18
27
  ```sh
19
28
  gauge status # the org at a glance
20
- gauge evals list # judged agent-experience measurements
21
- gauge preference list # Agent Preference prompts and measurements
29
+ gauge suite list # eval sets and Agent Preference prompts
22
30
  gauge models list # supported model × harness × provider targets
23
- gauge evals edit <id> --cadence weekly --samples 3 # when and how much it runs
24
- gauge evals run <id> # launch now (ORG-funded, credit-gated)
25
- gauge runs watch <run-id> # follow a run as the agent works
31
+ gauge suite edit eval <id> --cadence weekly --samples 3 # when and how much it runs
32
+ gauge run --eval <id> # launch now (ORG-funded, credit-gated)
33
+ gauge run watch <run-id> # follow a run as the agent works
26
34
  gauge stats rankings -o json # brand rankings (share of voice) as JSON
27
35
  gauge query -d ecosystem -m installs,install_rate -o json # a flexible report
28
36
  ```
29
37
 
38
+ `gauge instructions` prints a Gauge overview, CLI setup and usage rules, and
39
+ roughly 100 words of essential guidance per module. Each summary points to
40
+ `gauge instructions <topic>` for the full page. `gauge --help` points agents at
41
+ it, or write the file into the repo with `gauge instructions --install`.
42
+ The text ships inside the CLI, so it
43
+ never drifts from the commands it describes.
44
+
45
+ Use `gauge instructions run` before writing evals or starting runs: it covers
46
+ developer tasks, model selection, observable judging criteria, and a complete
47
+ migration example. Use `gauge instructions suite` to save eval sets and
48
+ preference prompts, and `gauge instructions session-analysis` to investigate
49
+ product choices, source exposure, and interventions, from individual traces to
50
+ larger cohorts.
51
+ Use `gauge instructions optimization` for the full optimization workflow.
52
+ `gauge instructions --list` shows available topics.
53
+ `gauge instructions run --install` installs the eval guidance as a local skill.
54
+
30
55
  Run `gauge --help` (or `gauge <command> --help`) for the full command set:
31
- evals, preference, models, personas, skills, mcp, repos, runs, experiments,
32
- batches, brands, tags, actions, dashboards, members, org, provider keys,
56
+ run, suite, models, personas, skills, mcp, repos, optimizations,
57
+ brands, tags, actions, dashboards, members, org, provider keys,
33
58
  billing, usage, stats, query, and more.
34
59
 
60
+ ## Run committed Markdown evals
61
+
62
+ File evals keep the task, run settings, and judged criteria in Git. Create
63
+ `gauge-evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
64
+
65
+ ```markdown
66
+ ---
67
+ criteria:
68
+ - name: Healthcheck responds
69
+ rubric: The endpoint returns HTTP 200 without authentication.
70
+ config:
71
+ repoUrl: https://github.com/acme/app
72
+ repoRef: main
73
+ agents:
74
+ - agent: CODEX_CLI
75
+ ---
76
+
77
+ Add a /health endpoint and run the relevant app checks.
78
+ ```
79
+
80
+ Check your setup, then commit the files before planning and running with Git inputs:
81
+
82
+ ```sh
83
+ gauge run --verify # local; includes uncommitted working-tree edits
84
+ gauge run --verify -o json # versioned global/per-input yes/no/unknown checklist
85
+ gauge run --dry-run
86
+ gauge run --yes
87
+ gauge run wait <request-id> -o json
88
+ ```
89
+
90
+ Cases default to `gauge-evals/**/*.md`. `--files` accepts quoted paths/globs and overrides
91
+ that convention for `--dry-run` and `run`. By default, these commands read committed Git
92
+ HEAD. With `--input`, they read your current files, including uncommitted edits.
93
+ Each case owns its agents, models, consumer repository/ref, Persona and criteria.
94
+ The optional `config.inputs` selects named inputs; all attach by default.
95
+
96
+ `verify` reads the repository's current files, including non-ignored untracked files,
97
+ without login, network access, a PR, built CI artifacts, or sandbox compute. It checks
98
+ configuration, case input references, Git input paths, and recognizable workflow
99
+ checkout/upload declarations. An input-only repository with no cases is valid setup.
100
+ Each attempted global/per-input check reports `yes` (established locally), `no`
101
+ (a definite problem), or `unknown` (a specific static question could not be resolved,
102
+ such as an expression-based artifact name or a referenced reusable workflow).
103
+ Missing optional configuration and zero cases are informational findings, excluded
104
+ from checklist counts. Eligible workflow triggers are checked as declarations;
105
+ event filters and runtime conditions are not evaluated against a candidate PR.
106
+ Live input delivery, artifact contents, sandbox execution, App activation, and
107
+ account settings are outside this command's scope and do not create unknown rows.
108
+ Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
109
+ No workflow code is executed.
110
+
111
+ `gauge run --verify -o json` writes exactly one JSON object to stdout, including on
112
+ validation failure. The versioned report provides:
113
+
114
+ - `version: 1` and `mode: "local-static"` identify the report contract and scope.
115
+ - `status` is `failed` if any check is `no`, otherwise `incomplete` if any check is
116
+ `unknown`, otherwise `passed`. It describes static checks only.
117
+ - `summary` contains `yes`, `no`, and `unknown` counts.
118
+ - `global` and `inputs[].checks` contain checks with stable `id`, `state`, and
119
+ explanatory `detail` fields; `inputs[].name` identifies each input.
120
+ - `facts.configFilePresent` and `facts.caseCount` expose discovered facts without
121
+ parsing prose. A `null` value means inspection did not reach that fact; zero cases
122
+ is distinct from case discovery being skipped or failing.
123
+ - `info` contains informational findings with stable `id` and `detail` fields.
124
+ - `notChecked` lists excluded capabilities as machine-readable identifiers.
125
+
126
+ For example, an agent can select unresolved checks with:
127
+
128
+ ```sh
129
+ gauge run --verify -o json | jq '{status, facts, global: [.global[] | select(.state != "yes")], inputs: [.inputs[] | {name, checks: [.checks[] | select(.state != "yes")]}]}'
130
+ ```
131
+
132
+ An optional `gauge.json` declares inputs produced by your normal CI or committed
133
+ alongside your product:
134
+
135
+ ```json
136
+ {
137
+ "version": 2,
138
+ "org": "my-org",
139
+ "inputs": {
140
+ "cli": {
141
+ "type": "npm-package",
142
+ "source": { "type": "github-actions", "workflow": ".github/workflows/package.yml" },
143
+ "path": "*.tgz",
144
+ "install": "environment"
145
+ },
146
+ "docs": {
147
+ "type": "files",
148
+ "source": { "type": "git" },
149
+ "paths": ["docs/**/*.md"]
150
+ }
151
+ }
152
+ }
153
+ ```
154
+
155
+ Gauge waits for your PR build and tests its uploaded artifacts. Keep the workflow's
156
+ existing checkout settings, including GitHub's default merge checkout. Omit
157
+ `source.artifact` when the workflow uploads one artifact, or select it by name.
158
+
159
+ For automatic checks, connect the Gauge GitHub App with repository access and
160
+ Actions read permission. Set `org` and enable the repository and target branches
161
+ in **Settings → GitHub**. Waiting for inputs starts no sessions and spends no
162
+ session credits.
163
+ Fork pull requests are disabled by default. Check **Allow fork pull requests** to let
164
+ workspace members authorize an evaluation from its GitHub check. Authorization
165
+ covers one PR revision and uses workspace credits; new revisions and explicit
166
+ reruns require fresh approval. Configured inputs determine when an authorized
167
+ evaluation is ready to run.
168
+ A failed run or absent/expired artifact returns an actionable input failure.
169
+ Absent runs have a five-minute discovery grace period; known running or queued
170
+ CI can continue past an hour. Abandoned requests expire after seven days. Reruns retain captured bytes. CI metadata reports a revision; it
171
+ does not independently prove which checkout the workflow compiled.
172
+
173
+ To run automatic PR checks only when relevant files change, add `checks` to
174
+ `gauge.json`:
175
+
176
+ ```json
177
+ "checks": {
178
+ "paths": ["agents/**", "docs.json", "snippets/**"]
179
+ }
180
+ ```
181
+
182
+ Paths are repository-relative, case-sensitive globs; matching any pattern runs
183
+ the checks. `**` matches nested directories, including hidden files. Use 1–32
184
+ patterns; exclusions (`!`) are not supported. Changes to `gauge.json` or your
185
+ configured case paths always count. Deleted files and both names of renamed
186
+ files are included. Without root `checks`, every eligible PR update reaches case
187
+ selection.
188
+
189
+ With CLI 0.16.0 or later, a case can narrow its own automatic PR triggers in its
190
+ Markdown frontmatter (alongside its other `config` settings):
191
+
192
+ ```yaml
193
+ config:
194
+ agents:
195
+ - agent: CODEX_CLI
196
+ checks:
197
+ paths: ["cli/**", "packages/shared/**"]
198
+ ```
199
+
200
+ Root `checks.paths`, when present, must match first. The App then runs matching
201
+ cases and all cases without a filter. Changes to `gauge.json` select every case;
202
+ editing a case selects that case regardless of its filter. If no cases match,
203
+ the check is skipped before preparing inputs or reserving credits. `config.inputs`
204
+ only chooses inputs; it does not define change triggers.
205
+
206
+ The first check uses the PR diff. After a passing evaluation, later updates
207
+ compare against that evaluated commit, so unrelated follow-up edits can skip.
208
+ A failed or unfinished evaluation, a changed target branch, or an unavailable
209
+ or incomplete diff causes a run. Skipped checks start no sessions and reserve
210
+ no credits. GitHub's rerun action runs regardless of paths; explicit CLI runs
211
+ also ignore this automatic-check filter.
212
+
213
+ File inputs stay outside the consumer repository and their paths are provided to
214
+ the agent. npm packages install with ordinary npm using the VM's Node/npm runtime.
215
+ `install` defaults to `environment`: a separate tools directory with the package's
216
+ own declared commands on PATH, including login shells. `workspace` adds a consumer
217
+ application dependency and accepts libraries without commands. Transitive registry
218
+ dependency ranges remain live. Plan output shows the installation destination and,
219
+ when a previous matching preparation captured it, command metadata. Otherwise it
220
+ reports that commands will be determined during preparation; planning never downloads
221
+ archives or starts VMs.
222
+
223
+ Native executables use `type: "binary"`, with already unpacked files in a GitHub
224
+ Actions artifact (or small committed Git files):
225
+
226
+ ```json
227
+ "gh": {
228
+ "type": "binary",
229
+ "source": {
230
+ "type": "github-actions",
231
+ "workflow": ".github/workflows/package.yml",
232
+ "artifact": "gh-linux"
233
+ },
234
+ "paths": ["dist/**"],
235
+ "platform": "linux-x64",
236
+ "commands": { "gh": "dist/bin/gh" }
237
+ }
238
+ ```
239
+
240
+ Paths are relative to the uploaded artifact root. Use your normal PR build;
241
+ for `actions/upload-artifact@v7`, use `archive: true`. Include companion files in
242
+ `paths` and explicitly map 1–32 command names to ordinary selected files. Gauge
243
+ validates Linux x86-64 ELF executables in its isolated artifact processor, restores
244
+ executable permissions, preserves the bundle outside the consumer repository, and
245
+ exposes the commands on PATH, including login shells. A candidate command overrides
246
+ an installed product command with the same name; two inputs selected by the same
247
+ case cannot expose the same command (including captured npm environment commands).
248
+
249
+ The first version supports ELF executables and PIE binaries compatible with the
250
+ consumer Linux runtime. It does not unpack nested release archives, install OS
251
+ packages, alter library search paths, or run arbitrary installation/version-check
252
+ commands. Prefer a static binary or build against a compatible Linux runtime;
253
+ header validation cannot prove shared-library compatibility. Scripts, macOS,
254
+ Windows, and ARM executables are unsupported. A missing loader/library encountered
255
+ when the agent invokes the command appears in session execution evidence, not in
256
+ static validation. Harness/runtime names (including shells, Node/npm, Python, Git,
257
+ archive/setup utilities, and agent commands) are reserved; use a different alias
258
+ for such a candidate. `GAUGE_INPUTS` gives the preserved bundle path.
259
+
260
+ A Skill input has `type: "skill"` and `path: "skills/product"`, or a path to a ZIP,
261
+ `.tar.gz`, or `.tgz` whose root contains `SKILL.md`. Use `path: "."` for a Skill at
262
+ the source root. Gauge validates and extracts selected archives in isolation and
263
+ stages the Skill in the coding agent's native discovery location. File inputs use
264
+ `paths` to select committed files or files inside an Actions artifact.
265
+
266
+ Gauge tests the artifact uploaded by your PR's successful build. Keep your
267
+ existing checkout settings. The GitHub check links to the selected build;
268
+ rerun Gauge to test the latest build.
269
+
270
+ ### Run from your own CI or local build
271
+
272
+ With CLI 0.18.0 or later, supply built inputs directly to run without the GitHub App:
273
+
274
+ ```sh
275
+ gauge --org my-org run --input product-analytics=./dist/skill.zip --yes -o json > request.json
276
+ gauge --org my-org run wait "$(jq -er '.runRequestId' request.json)" --check -o json > results.json
277
+ ```
278
+
279
+ `--check` fails the job unless the verdict is `PASSED` or `MEASURED`. Without
280
+ it, `run wait` exits 0 once the request settles, whatever the verdict.
281
+
282
+ Set `GAUGE_API_TOKEN` as a CI secret and replace `my-org` with your workspace slug.
283
+ This can run immediately after your build in the same CI job.
284
+
285
+ Repeat `--input name=path` for each input. Skills accept a directory containing
286
+ `SKILL.md` or a ZIP/tar.gz archive; npm inputs accept a packed `.tgz`. Files and
287
+ binary inputs accept a directory, with configured paths relative to that directory.
288
+ Use a dedicated build directory: links are rejected, and common credential files
289
+ and `node_modules` are excluded. Each input is limited to 512 MiB and 20,000 files.
290
+
291
+ With `--input`, Gauge uses your current evaluation files, including uncommitted
292
+ edits. No Git checkout is required when all selected inputs are supplied locally.
293
+ Inputs without an override keep their configured source. You can omit `source`
294
+ in `gauge.json` to default to Git; the override replaces it for this run.
295
+ Website previews and private consumer repositories still require their configured
296
+ access. Public consumer repositories and the default workspace need no GitHub App.
297
+
298
+ `wait -o json` returns the verdict, session results, criterion explanations, and
299
+ session links. It exits 0 once the request settles, whatever the verdict. Add
300
+ `--check` to exit 1 unless the verdict is `PASSED` or `MEASURED`; the results
301
+ JSON is still the only output. Automatic PR checks continue to use configured
302
+ sources; disable those checks in Gauge's GitHub settings if CI owns the trigger.
303
+ Save `results.json` as a CI artifact even when the check fails.
304
+
305
+ ### Authenticated MCP servers
306
+
307
+ Connect your account in Gauge's MCP setup, completing OAuth in your browser when
308
+ required. Add the server's versioned reference to each case's `config.mcpRefs`.
309
+ Gauge manages credentials for the session; keep tokens out of prompts and Git.
310
+
311
+ ### Website previews
312
+
313
+ Website inputs use `type: "website"` and a canonical HTTPS origin in `url`.
314
+ The preview can come from `source: { "type": "github-deployment", "environment": "preview" }`
315
+ or a `github-pr-comment` source with an exact `author`, literal `bodyIncludes`
316
+ marker, and a readiness `pattern` with named `url` and `sha` captures. Gauge selects
317
+ candidate-specific evidence; timestamps alone cannot establish the revision.
318
+ Comments must belong to a unique open same-repository PR at the candidate head.
319
+ Previews must be public HTTPS origins, without authentication or path prefixes.
320
+ They are fetched live rather than snapshotted.
321
+
322
+ Editor schemas are published at
323
+ `https://agents.withgauge.com/schemas/gauge/v2.7.0/gauge.json` and
324
+ `https://agents.withgauge.com/schemas/gauge/v2.7.0/case.json`. Associate them in editor
325
+ settings; `gauge.json` does not accept a `$schema` field. These release URLs are
326
+ immutable. Update earlier pilot configs by moving workflows into input sources,
327
+ replacing `use` with input `type` and `path`/`paths`, and moving cases into
328
+ `gauge-evals/` (or setting `cases` explicitly).
329
+
330
+ `--org` overrides the committed organization. Root `cases` can change discovery;
331
+ root `version` defaults to 2. Saved evals use `gauge run --eval <id>`.
332
+
35
333
  ## Measurements own their configuration
36
334
 
37
335
  Every runnable thing carries its own complete run configuration — repository,
@@ -41,7 +339,7 @@ its own cadence and sample count. Nothing is shared, attached, or detached.
41
339
  An **eval** is one prompt against one configuration:
42
340
 
43
341
  ```sh
44
- gauge evals create --name "add a healthcheck" \
342
+ gauge suite create eval --name "add a healthcheck" \
45
343
  --text "Add a /health endpoint that returns 200 OK." \
46
344
  --criterion "Returns 200: the endpoint responds 200 with no auth" \
47
345
  --repo https://github.com/acme/api --ref main \
@@ -50,7 +348,7 @@ gauge evals create --name "add a healthcheck" \
50
348
  --cadence weekly --samples 3
51
349
  ```
52
350
 
53
- `gauge evals edit <id>` changes any of it later; omitted flags keep their
351
+ `gauge suite edit eval <id>` changes any of it later; omitted flags keep their
54
352
  stored value, and each `--clear-*` flag empties one field. To measure the same
55
353
  prompt against a second repository or roster, copy the eval in the app and edit
56
354
  the copy — one eval is always one configuration.
@@ -62,17 +360,17 @@ the prompt's settings, so one prompt measures the same question across several
62
360
  repositories and personas at once:
63
361
 
64
362
  ```sh
65
- gauge preference create --name "CLI deployment preference" \
363
+ gauge suite create preference --name "CLI deployment preference" \
66
364
  --kind head-to-head --brand-a acme --brand-b globex \
67
365
  --text "Which CLI should I use to deploy?" \
68
366
  --agent claude-code --cadence daily \
69
367
  --repo https://github.com/acme/api # first scenario
70
368
 
71
- gauge preference edit <promptId> --agent codex --samples 3 --cadence weekly
72
- gauge preference scenarios list <promptId>
73
- gauge preference scenarios add <promptId> --persona <profileId>
74
- gauge preference scenarios add <promptId> --repo https://github.com/acme/web --persona <profileId>
75
- gauge preference scenarios rm <promptId> --scenario <scenarioId>
369
+ gauge suite edit preference <promptId> --agent codex --samples 3 --cadence weekly
370
+ gauge suite scenarios list <promptId>
371
+ gauge suite scenarios add <promptId> --persona <profileId>
372
+ gauge suite scenarios add <promptId> --repo https://github.com/acme/web --persona <profileId>
373
+ gauge suite scenarios rm <promptId> --scenario <scenarioId>
76
374
  ```
77
375
 
78
376
  A prompt with zero scenarios is a valid definition — it simply never comes due.
@@ -84,14 +382,114 @@ run-settings flags on `create` and the org's default roster applies.
84
382
  `--cadence none` is "manual": the measurement stops coming due and runs only
85
383
  when you launch it. There is no separate pause or enabled flag, in the CLI or
86
384
  the app. `--cadence daily|weekly|monthly` puts it back on a rhythm, and
87
- `preference get` reads a manual prompt back as `cadence: manual`.
385
+ `suite get preference <id>` reads a manual prompt back as `cadence: manual`.
88
386
 
89
- `gauge evals run <id>` and `gauge preference run <id>` are the separate
387
+ `gauge run --eval <id>` and `gauge run --preference <id>` are the separate
90
388
  immediate launches. They use the stored configuration and ask before spending
91
389
  (`--yes` skips the prompt for scripts).
92
- `gauge preference run <id> --scenario <scenarioId>` (repeatable) runs a subset;
390
+ `gauge run --preference <id> --scenario <scenarioId>` (repeatable) runs a subset;
93
391
  omit it and every scenario runs. Neither launch touches the cadence clock.
94
392
 
393
+ ## Optimizations
394
+
395
+ Start by finding runs that encountered the surface you want to improve:
396
+
397
+ ```sh
398
+ gauge instructions optimization
399
+ gauge optimizations baselines --eval ev_1 -o json
400
+ gauge run insight run_1 -o json
401
+ gauge run get run_1 --logs -o json
402
+ gauge optimizations start --eval ev_1 --from-run run_1 --yes
403
+ gauge optimizations watch cl_1
404
+ ```
405
+
406
+ `baselines` returns observed owned surfaces grouped by run, source evidence,
407
+ judging status, eligibility (including the baseline's complete run set), and a
408
+ `nextCursor` for `--cursor`. It uses the UI baseline picker's compatibility rules
409
+ and recent 100-batch eligibility window. Missing evidence is unknown; the list
410
+ is an analysis summary, not an exhaustive inventory of every trace event.
411
+ `--from-run` inherits the eligible baseline's harness/model and reuses its runs.
412
+ A stale choice fails instead of silently buying a new baseline. `--target` is
413
+ an alternative when you need to establish a baseline; do not combine it with
414
+ `--from-run`. Use `--preference` instead of `--eval` for preference prompts.
415
+
416
+ ```sh
417
+ gauge optimizations authoring cl_1 > context.json
418
+ gauge optimizations page cl_1 --url https://docs.acme.dev/auth -o json
419
+ # Author round.json using context.revision and the observed baseline evidence.
420
+ gauge optimizations round preview cl_1 --file round.json -o json
421
+ gauge optimizations round save cl_1 --file round.json --digest <previewDigest>
422
+ gauge optimizations launch cl_1 --yes
423
+ gauge optimizations watch cl_1
424
+ gauge optimizations trial cl_1 R1.1 -o json
425
+ gauge optimizations adopt cl_1 R1.1 --yes
426
+ gauge optimizations handoff cl_1
427
+ gauge optimizations complete cl_1 --yes
428
+ ```
429
+
430
+ A round is `{revision, trials: [{trial: {title, hypothesis, lever, intervention}}]}`.
431
+ Docs `REWRITE_DOCS` proposals supply `after.pages` with URL and exact `edits`
432
+ (`find`/`replace`), or a complete `body` and `rewriteReason`. Check the page's
433
+ `source`: `scrape` means current fetched content, not the captured response.
434
+
435
+ For CLI trials, inspect `cliIntervention.guidance` and `targets` in authoring
436
+ and use `gauge run get <id> --logs` for the cited command evidence. `cli-change --file change.json`
437
+ accepts v3 `{schemaVersion:3, target, rules, shippingIntent}` and returns a sealed
438
+ bundle to put in `CLI_INTERVENTION.after`. It does not save or launch. The old
439
+ catalog/invocation-ID workflow is retired. Skills use `skill-files` and the
440
+ complete edited file bundle.
441
+
442
+ A checkpoint is a saved state of a baseline run. Resuming it avoids repeating
443
+ earlier work. Gauge automatically selects a compatible checkpoint before the
444
+ affected interaction for docs and CLI candidates, or starts fresh with a reason.
445
+ You do not select a checkpoint; omit checkpoint choices and fork flags. Skills
446
+ start fresh. Inspect
447
+ `runs[].candidateExecution` for actual starting mode and `candidateDelivery`
448
+ for exposure, observed artifacts, and evidence completeness. A score improvement
449
+ does not prove the change was exercised. Report sample counts and failed sessions.
450
+
451
+ Read `nextAction` on optimization detail; `watch` exits 4 when input is needed.
452
+ Round save replaces pending drafts but never launches. Use `--yes` for approved
453
+ spending and `trials add --idempotency-key` when retrying single-trial writes.
454
+ Adoption selects a winner without changing the baseline or deploying the change;
455
+ `handoff` supplies instructions to implement it in your source repository.
456
+
457
+ ## Compare runs
458
+
459
+ `gauge run compare` answers before/after questions in one call. Each side takes
460
+ repeatable `key=value` selectors: `request=`, `batch=`, `eval=`, `revision=`,
461
+ `experiment=`, `trial=`, `agent=`, `model=`, `since=`, and `until=`.
462
+
463
+ ```sh
464
+ gauge run compare --baseline request=<id> --candidate request=<id>
465
+ gauge run compare --baseline eval=<id> --baseline until=2026-10-01 \
466
+ --candidate eval=<id> --candidate since=2026-10-01
467
+ ```
468
+
469
+ It prints the pass rate, each criterion's pass rate, and median turns, cost,
470
+ and duration, for all runs and per agent and model. `-o json` returns the same
471
+ data with deltas.
472
+
473
+ Each run has an `evalRevision`: a hash of its prompt, repository, persona,
474
+ agent context, and criteria. `gauge suite edit` changes the revision of new
475
+ runs. Each case has its own revision. `gauge run list` shows it, and `compare`
476
+ warns when one eval or case ran under more than one revision; add
477
+ `revision=<hash>` to pick one. `compare` also warns when a side matched no
478
+ runs, has runs in progress, or has runs without a verdict.
479
+
480
+ Optimization trial runs test a changed configuration, so `eval=` and other
481
+ selectors leave them out. Add `trial=<trialId>` to compare one trial's runs,
482
+ including runs it reused. `gauge run list` shows the trial key and id.
483
+ `compare` prints the trial behind each side's trial runs. `gauge run list` and
484
+ `gauge run get` show a run's `optimizationTrial`.
485
+
486
+ To try other agents or sample counts once, without editing the suite, pass
487
+ `--agent` and `--samples` to `gauge run --eval <id>` or `--preference <id>`.
488
+ The stored configuration stays unchanged.
489
+
490
+ `gauge run status <request-id>` prints a launch request's progress and
491
+ per-session results once, without waiting.
492
+
95
493
  ## Models, reports, and scripting
96
494
 
97
495
  Use `gauge models list` before pinning models. It reads the organization's live
@@ -106,8 +504,93 @@ and effective capabilities. Open harnesses use `--agent pi:<model>` and
106
504
 
107
505
  Every read supports `-o json` for scripting. `GAUGE_API_TOKEN` overrides
108
506
  stored credentials (useful in CI). Exit codes: `0` ok · `1` error ·
109
- `2` usage · `3` billing-blocked.
507
+ `2` usage · `3` billing-blocked · `4` needs your input (`optimizations watch`).
110
508
 
111
509
  ## Support
112
510
 
113
511
  Questions or problems: [support@withgauge.com](mailto:support@withgauge.com).
512
+
513
+
514
+ ### Git-authored Agent Preference
515
+
516
+ A committed Markdown case may declare `preference` instead of `criteria`.
517
+ Use exactly one; the Markdown body is always the coding agent's user request.
518
+
519
+ ```markdown
520
+ ---
521
+ preference:
522
+ kind: open-ended
523
+ config:
524
+ agents:
525
+ - agent: CODEX_CLI
526
+ sampleCount: 5
527
+ inputs: [docs]
528
+ ---
529
+
530
+ I need product analytics for a small SaaS application.
531
+ We need funnels, retention analysis, and EU data residency.
532
+ What should we use?
533
+ ```
534
+
535
+ Here `docs` names an input declared in `gauge.json`. Existing repository/persona
536
+ settings and automatic path filters apply unchanged. Add `branded: true` when an
537
+ open-ended prompt names a brand. For a comparison, use:
538
+
539
+ ```yaml
540
+ preference:
541
+ kind: head-to-head
542
+ brandA: PostHog
543
+ brandB: Amplitude
544
+ ```
545
+
546
+ Write both brands in the prompt itself; metadata is only for analysis and never
547
+ rewrites the task. `gauge run --verify`, `--dry-run`, and `gauge run` support mixed experience
548
+ and preference suites. These fields require the CLI version containing this
549
+ feature; CLI 0.16.0 and earlier reject them.
550
+
551
+ Preference uses the existing app analyst: ranked picks for open-ended cases,
552
+ and a winner (or neither) for head-to-head. Results are advisory. Preference-only
553
+ GitHub checks finish neutral; experience criteria still gate mixed suites.
554
+ `gauge run wait --check` exits successfully for `MEASURED` or `PASSED`;
555
+ input, execution, and analysis errors exit 1. Results describe the
556
+ candidate revision and do not infer a base/head delta.
557
+
558
+ ## Local process lifecycle
559
+
560
+ `gauge run --local` runs in the foreground. Ctrl+C (or SIGTERM) stops the
561
+ queue, cancels active sessions, waits for cleanup, and exits with code 130.
562
+ Partial results and traces remain under `.gauge/runs`. Successful sessions
563
+ remove their containers and temporary credentials; the Docker image stays cached.
564
+
565
+ Use `gauge run --detach --local` to keep a batch running after the CLI exits.
566
+ A detached supervisor drains the queue and exits when the batch settles. The
567
+ launch prints its batch ID and these commands, to run from the same repository:
568
+
569
+ ```sh
570
+ gauge run status --batch <batch-id> --local
571
+ gauge run wait --batch <batch-id> --local
572
+ gauge run cancel --batch <batch-id> --local
573
+ ```
574
+
575
+ `gauge run wait --batch <batch-id> --local` blocks until the batch settles
576
+ (`completed`, `canceled`, `failed`, or `interrupted`), then prints each
577
+ session's state and the next step. It exits 0 once settled; `--check` exits 1
578
+ unless every session completed.
579
+
580
+ `gauge run status <run-id> --local` inspects a single session;
581
+ `gauge run cancel <run-id> --local` cancels only that run. All support `-o json`.
582
+ Batch cancellation is persisted before workers are contacted, including during
583
+ startup. A pending cancellation returns nonzero; inspect status before retrying.
584
+
585
+ After an interruption, `gauge cleanup --local` retries cleanup for recorded
586
+ interrupted sessions or sessions with incomplete cleanup in this repository.
587
+ It preserves evidence and recovered work, and makes recovered results visible
588
+ in `gauge run list`. It leaves live sessions, other repositories, unrelated
589
+ containers, and cached images alone. Recovery checks Gauge's container label
590
+ and workspace mount before removing a remaining container. Uncertain worker
591
+ ownership is reported rather than signaling a potentially reused PID.
592
+
593
+ Local workers have a 30-minute turn deadline and a 60-second supervisor
594
+ response deadline, so losing a supervisor cannot leave an idle worker forever.
595
+ Cleanup cannot discover sessions whose `.gauge` records were deleted; keep
596
+ those records until cleanup has succeeded.
@@ -0,0 +1,25 @@
1
+ import {
2
+ batchDirectory,
3
+ runLocal
4
+ } from "./chunk-X6GOZUKP.js";
5
+ import "./chunk-VZZMTTKV.js";
6
+
7
+ // src/batchWorker.ts
8
+ import { readFile, rm, writeFile } from "fs/promises";
9
+ import { join } from "path";
10
+ var request = JSON.parse(await readFile(process.argv[2], "utf8"));
11
+ await rm(process.argv[2]);
12
+ try {
13
+ await runLocal(request.definition, [], request.gaugeDir, () => {
14
+ }, {
15
+ batchId: request.batchId
16
+ });
17
+ } catch (error) {
18
+ console.error(error);
19
+ await writeFile(
20
+ join(batchDirectory(request.gaugeDir, request.batchId), "error.txt"),
21
+ String(error),
22
+ { mode: 384 }
23
+ );
24
+ process.exitCode = 1;
25
+ }