@withgauge/cli 0.8.0 → 0.9.0-internal.20261019.8166718

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,45 +1,640 @@
1
1
  # @withgauge/cli
2
2
 
3
3
  `gauge` — the [Gauge](https://agents.withgauge.com) command-line interface.
4
- Manage your eval sets, Agent Preference prompts, run presets, and agent runs,
5
- and watch runs live from your terminal.
4
+ Manage your eval sets, Agent Preference prompts, and agent runs, and watch runs
5
+ live from your terminal.
6
6
 
7
7
  ```sh
8
8
  npm install -g @withgauge/cli
9
- gauge onboard # sign in, create a workspace, seed starter measurements
9
+ gauge update # update the CLI to the latest published version
10
+ gauge onboard # sign in and create a workspace (account only)
10
11
  gauge auth login # (or just sign in)
11
12
  gauge orgs use <org> # set your default organization
13
+ gauge init # set up evals in this repo: hosted when signed in, else local
14
+ gauge init --hosted # switch a local repo to hosted Gauge (after gauge onboard)
12
15
  ```
13
16
 
14
17
  Requires Node.js ≥ 22.12.
15
18
 
19
+ ## Local Docker runtime
20
+
21
+ Local runs need Docker running Linux containers and an agent credential. Linux
22
+ is the container OS, not a requirement for your host: Docker Desktop on macOS
23
+ provides a Linux daemon and shares the local files used by sessions.
24
+ Install the runtime without a Gauge account or a source checkout:
25
+
26
+ ```sh
27
+ gauge local runtime pull
28
+ gauge local runtime inspect -o json
29
+ gauge run --local --dry-run
30
+ ```
31
+
32
+ Each CLI release pins its tested public `ghcr.io/gauge-sh/gauge-runtime` image by
33
+ immutable digest. Inspection reports the reference, local image ID, registry
34
+ digest, contract, platform, and readiness errors. `ready` covers local image
35
+ metadata; agent credential checks happen separately. Neither inspection nor a
36
+ dry run downloads an image. `pull` explicitly downloads the supported image;
37
+ previously pulled images can be reused offline (agent calls still need network
38
+ access). Pull progress goes to stderr so JSON output remains parseable.
39
+
40
+ The runtime includes native `linux/amd64` and `linux/arm64` images. Docker selects
41
+ the matching image on Intel/AMD or Apple Silicon/ARM machines. No architecture
42
+ emulation is needed. Docker must be able to bind-mount this machine's temporary
43
+ and input directories; an unrelated remote Docker server cannot access them.
44
+ On Windows, run the CLI inside WSL because Gauge uses Unix sockets and UID/GID.
45
+
46
+ ## Local agent credentials
47
+
48
+ Local provider credentials are separate from a Gauge account and hosted credential
49
+ brokering. Use native harness logins or configure a named environment reference:
50
+
51
+ ```sh
52
+ gauge local profiles set work --agent CODEX_CLI --provider openai --auth api_key --env TEAM_OPENAI_KEY
53
+ gauge local profiles use work
54
+ gauge local profiles inspect work -o json
55
+ gauge run --local --profile work --dry-run -o json
56
+ ```
57
+
58
+ `--env` takes the variable's name, never its secret value. Profiles store only
59
+ references; list and inspect output do not expose credentials. Update a profile
60
+ with `set ... --replace` or remove it with `remove <profile>` without deleting
61
+ the native login. Local readiness checks do not verify provider acceptance or
62
+ model access. Use `gauge local profiles --help` for profile commands and
63
+ `gauge instructions --local` for the local workflow.
64
+
65
+ ## Anonymous CLI telemetry
66
+
67
+ The installed CLI collects command usage by default to help improve Gauge.
68
+ Opt out for your user account or for a single process:
69
+
70
+ ```sh
71
+ gauge telemetry disable
72
+ gauge telemetry status
73
+ gauge telemetry enable
74
+ GAUGE_TELEMETRY_DISABLED=1 gauge run --verify
75
+ ```
76
+
77
+ The environment variable overrides the saved preference. These controls work
78
+ without an account or network access and support `-o json`. They apply to CLI
79
+ command telemetry; authenticated Gauge API requests have separate server-side
80
+ usage analytics.
81
+
82
+ Each `cli_command_invoked` event contains only the canonical command name
83
+ (for example, `run list`), its UTC invocation timestamp, CLI version, and a
84
+ random installation UUID. Arguments, flag values, file paths, prompts, file
85
+ contents, environment variables, errors, and account identifiers are excluded.
86
+ The UUID is independent of login and is never linked to your Gauge account.
87
+ This is pseudonymous usage tracking: an installation can be recognized across
88
+ commands, but no personal identity is sent. HTTP requests necessarily expose a
89
+ source IP to the receiving infrastructure; event properties mask IP addresses,
90
+ disable GeoIP enrichment, and disable person-profile creation.
91
+
92
+ Preferences live in `~/.config/gauge/telemetry.json` and the UUID in
93
+ `~/.config/gauge/telemetry-id` (respecting `XDG_CONFIG_HOME`). They are shared
94
+ across repositories, independently of each repository's `.gauge/` results.
95
+ Deleting `telemetry-id` resets the identity on the next collected command.
96
+ Consumers are installations, not necessarily individual people. Session
97
+ analysis can group an installation's events using a 30-minute inactivity gap;
98
+ the CLI does not generate a different identity for each invocation.
99
+
100
+ Collection occurs when a parsed command is about to run, including commands
101
+ whose handlers subsequently fail. Help, version, parsing errors, telemetry
102
+ controls, and source development runs are excluded. A detached worker handles
103
+ all telemetry storage and network access, with a two-second request timeout
104
+ and a three-second worker lifetime. Commands never wait for delivery, and
105
+ failures are silent. There are no retries or offline queue, so events may be
106
+ lost. Corrupt or unreadable preferences suppress collection until repaired.
107
+
108
+ Events go to Gauge's existing PostHog project using its public ingestion key.
109
+ The [capture API](https://posthog.com/docs/api/capture) and
110
+ [anonymous event controls](https://posthog.com/docs/product-analytics/capture-events#advanced-anonymous-vs-identified-events)
111
+ describe the transport and person-profile behavior.
112
+
113
+ ## Local or hosted
114
+
115
+ Each repo runs in one mode: `mode` in gauge.json. Without it, Gauge uses hosted
116
+ when you are signed in, else local. `--local` and `--hosted` override it.
117
+
118
+ | Workflow | Local | Hosted |
119
+ |---|---|---|
120
+ | Skill, npm package (CLI or SDK), and `files` inputs | yes | yes |
121
+ | Docs supplied as `files` inputs | yes | yes |
122
+ | Eval and Agent Preference cases in `gauge-evals/` | yes (you read the sessions) | yes (judged) |
123
+ | `run`, `run list`, `run compare --local`, `models list`, `cleanup`, `feedback` | yes | yes |
124
+ | Binary, website, and MCP inputs | no | yes |
125
+ | Saved suites, scenarios, personas, PR checks | no | yes |
126
+ | Optimizations (managed docs rewriting) | no | yes |
127
+ | Stored credentials for MCP servers and connections | no | yes |
128
+ | Stats, queries, dashboards, org settings | no | yes |
129
+
130
+ Local runs need Docker and the agent's own login (`claude` or
131
+ `ANTHROPIC_API_KEY` for Claude Code; `codex login` for Codex, Pi, and
132
+ OpenCode). They do not need a Gauge account. Hosted runs need a Gauge account
133
+ (`gauge onboard`) and use credits. In local mode, a hosted-only command says so
134
+ and prints the next step.
135
+
136
+ For coding agents, we recommend the
137
+ [`gauge-agents` skill](https://github.com/gauge-sh/gauge-skills/tree/main/gauge-agents).
138
+ It makes Gauge CLI guidance and repository setup workflows discoverable in future
139
+ sessions. Install it separately from repo setup, running from your project root:
140
+
141
+ ```sh
142
+ gauge install-skill
143
+ ```
144
+
145
+ Select your coding agent and installation scope in the installer. Project scope
146
+ adds skill files to this repo; global scope makes the skill available across projects.
147
+ The command runs the pinned `skills@1.5.0` installer through npx and forwards its
148
+ prompts and output. For automated installation after choosing an agent:
149
+
150
+ ```sh
151
+ gauge install-skill --agent claude-code --yes # this project
152
+ gauge install-skill --agent codex --global --yes # across projects
153
+ ```
154
+
155
+ Repeat `--agent` to target multiple agents. `--yes` requires an explicit agent
156
+ target; `--copy` copies files instead of creating symlinks.
157
+
16
158
  ## Usage
17
159
 
18
160
  ```sh
19
161
  gauge status # the org at a glance
20
- gauge evals list # judged agent-experience measurements
21
- gauge preference list # Agent Preference prompts and measurements
22
- gauge presets list # run configurations (repo × persona × agents × skills × MCP)
162
+ gauge suite list # eval sets and Agent Preference prompts
23
163
  gauge models list # supported model × harness × provider targets
24
- gauge evals attach <id> --preset <presetId> # schedule a preset on an eval (prints the bill first)
25
- gauge evals run <id> # launch now (ORG-funded, credit-gated)
26
- gauge runs watch <run-id> # follow a run as the agent works
164
+ gauge suite edit eval <id> --cadence weekly --samples 3 # when and how much it runs
165
+ gauge run --eval <id> # launch now (ORG-funded, credit-gated)
166
+ gauge run watch <run-id> # follow a run as the agent works
27
167
  gauge stats rankings -o json # brand rankings (share of voice) as JSON
28
168
  gauge query -d ecosystem -m installs,install_rate -o json # a flexible report
29
169
  ```
30
170
 
171
+ `gauge instructions` prints a Gauge overview, CLI setup and usage rules, and
172
+ roughly 100 words of essential guidance per module. Each summary points to
173
+ `gauge instructions <topic>` for the full page. `gauge --help` points agents at
174
+ it, or write the file into the repo with `gauge instructions --install`.
175
+ The text ships inside the CLI, so it
176
+ never drifts from the commands it describes.
177
+
178
+ Use `gauge instructions run` before writing evals or starting runs: it covers
179
+ developer tasks, model selection, observable judging criteria, and a complete
180
+ migration example. Use `gauge instructions suite` to save eval sets and
181
+ preference prompts, and `gauge instructions session-analysis` to investigate
182
+ product choices, source exposure, and interventions, from individual traces to
183
+ larger cohorts.
184
+ Use `gauge instructions optimization` for the full optimization workflow.
185
+ `gauge instructions --list` shows available topics.
186
+ `gauge instructions run --install` installs the eval guidance as a local skill.
187
+
31
188
  Run `gauge --help` (or `gauge <command> --help`) for the full command set:
32
- evals, preference, presets, models, personas, skills, mcp, cycles, repos, runs,
33
- experiments, batches, brands, tags, actions, impl, dashboards, members, org,
34
- provider keys, billing, usage, stats, query, and more.
35
-
36
- **Run presets** are the unit of configuration: an eval set or preference
37
- prompt runs whatever presets are attached to it, every cycle. `gauge presets
38
- create --name api-main --repo https://github.com/acme/api --agent
39
- claude-code:claude-opus-4-8 --skill acme-cli@1.2.0` authors one;
40
- `gauge evals attach` / `gauge preference attach` schedules it. Attaching
41
- spends money on a schedule, so `attach` and `run` print the per-cycle run
42
- arithmetic and ask before acting (`--yes` skips the prompt for scripts).
189
+ run, suite, models, personas, skills, mcp, repos, optimizations,
190
+ brands, tags, actions, dashboards, members, org, provider keys,
191
+ billing, usage, stats, query, and more.
192
+
193
+ ## Run committed Markdown evals
194
+
195
+ File evals keep the task, run settings, and judged criteria in Git. Create
196
+ `gauge-evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
197
+
198
+ ```markdown
199
+ ---
200
+ criteria:
201
+ - name: Healthcheck responds
202
+ rubric: The endpoint returns HTTP 200 without authentication.
203
+ config:
204
+ repoUrl: https://github.com/acme/app
205
+ repoRef: main
206
+ agents:
207
+ - agent: CODEX_CLI
208
+ ---
209
+
210
+ Add a /health endpoint and run the relevant app checks.
211
+ ```
212
+
213
+ Check your setup, then commit the files before planning and running with Git inputs:
214
+
215
+ ```sh
216
+ gauge run --verify # local; includes uncommitted working-tree edits
217
+ gauge run --verify -o json # versioned global/per-input yes/no/unknown checklist
218
+ gauge run --dry-run
219
+ gauge run --yes
220
+ gauge run wait <request-id> -o json
221
+ ```
222
+
223
+ Cases default to `gauge-evals/**/*.md`. `--files` accepts quoted paths/globs and overrides
224
+ that convention for `--dry-run` and `run`. By default, these commands read committed Git
225
+ HEAD. With `--input`, they read your current files, including uncommitted edits.
226
+ Each case owns its agents, models, consumer repository/ref, Persona and criteria.
227
+ The optional `config.inputs` selects named inputs; all attach by default.
228
+
229
+ `verify` reads the repository's current files, including non-ignored untracked files,
230
+ without login, network access, a PR, built CI artifacts, or sandbox compute. It checks
231
+ configuration, case input references, Git input paths, and recognizable workflow
232
+ checkout/upload declarations. An input-only repository with no cases is valid setup.
233
+ Each attempted global/per-input check reports `yes` (established locally), `no`
234
+ (a definite problem), or `unknown` (a specific static question could not be resolved,
235
+ such as an expression-based artifact name or a referenced reusable workflow).
236
+ Missing optional configuration and zero cases are informational findings, excluded
237
+ from checklist counts. Eligible workflow triggers are checked as declarations;
238
+ event filters and runtime conditions are not evaluated against a candidate PR.
239
+ Live input delivery, artifact contents, sandbox execution, App activation, and
240
+ account settings are outside this command's scope and do not create unknown rows.
241
+ Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
242
+ No workflow code is executed.
243
+
244
+ `gauge run --verify -o json` writes exactly one JSON object to stdout, including on
245
+ validation failure. The versioned report provides:
246
+
247
+ - `version: 1` and `mode: "local-static"` identify the report contract and scope.
248
+ - `status` is `failed` if any check is `no`, otherwise `incomplete` if any check is
249
+ `unknown`, otherwise `passed`. It describes static checks only.
250
+ - `summary` contains `yes`, `no`, and `unknown` counts.
251
+ - `global` and `inputs[].checks` contain checks with stable `id`, `state`, and
252
+ explanatory `detail` fields; `inputs[].name` identifies each input.
253
+ - `facts.configFilePresent` and `facts.caseCount` expose discovered facts without
254
+ parsing prose. A `null` value means inspection did not reach that fact; zero cases
255
+ is distinct from case discovery being skipped or failing.
256
+ - `info` contains informational findings with stable `id` and `detail` fields.
257
+ - `notChecked` lists excluded capabilities as machine-readable identifiers.
258
+
259
+ For example, an agent can select unresolved checks with:
260
+
261
+ ```sh
262
+ gauge run --verify -o json | jq '{status, facts, global: [.global[] | select(.state != "yes")], inputs: [.inputs[] | {name, checks: [.checks[] | select(.state != "yes")]}]}'
263
+ ```
264
+
265
+ An optional `gauge.json` declares inputs produced by your normal CI or committed
266
+ alongside your product:
267
+
268
+ ```json
269
+ {
270
+ "version": 2,
271
+ "org": "my-org",
272
+ "inputs": {
273
+ "cli": {
274
+ "type": "npm-package",
275
+ "source": { "type": "github-actions", "workflow": ".github/workflows/package.yml" },
276
+ "path": "*.tgz",
277
+ "install": "environment"
278
+ },
279
+ "docs": {
280
+ "type": "files",
281
+ "source": { "type": "git" },
282
+ "paths": ["docs/**/*.md"]
283
+ }
284
+ }
285
+ }
286
+ ```
287
+
288
+ Gauge waits for your PR build and tests its uploaded artifacts. Keep the workflow's
289
+ existing checkout settings, including GitHub's default merge checkout. Omit
290
+ `source.artifact` when the workflow uploads one artifact, or select it by name.
291
+
292
+ For automatic checks, connect the Gauge GitHub App with repository access and
293
+ Actions read permission. Set `org` and enable the repository and target branches
294
+ in **Settings → GitHub**. Waiting for inputs starts no sessions and spends no
295
+ session credits.
296
+ Fork pull requests are disabled by default. Check **Allow fork pull requests** to let
297
+ workspace members authorize an evaluation from its GitHub check. Authorization
298
+ covers one PR revision and uses workspace credits; new revisions and explicit
299
+ reruns require fresh approval. Configured inputs determine when an authorized
300
+ evaluation is ready to run.
301
+ A failed run or absent/expired artifact returns an actionable input failure.
302
+ Absent runs have a five-minute discovery grace period; known running or queued
303
+ CI can continue past an hour. Abandoned requests expire after seven days. Reruns retain captured bytes. CI metadata reports a revision; it
304
+ does not independently prove which checkout the workflow compiled.
305
+
306
+ To run automatic PR checks only when relevant files change, add `checks` to
307
+ `gauge.json`:
308
+
309
+ ```json
310
+ "checks": {
311
+ "paths": ["agents/**", "docs.json", "snippets/**"]
312
+ }
313
+ ```
314
+
315
+ Paths are repository-relative, case-sensitive globs; matching any pattern runs
316
+ the checks. `**` matches nested directories, including hidden files. Use 1–32
317
+ patterns; exclusions (`!`) are not supported. Changes to `gauge.json` or your
318
+ configured case paths always count. Deleted files and both names of renamed
319
+ files are included. Without root `checks`, every eligible PR update reaches case
320
+ selection.
321
+
322
+ With CLI 0.16.0 or later, a case can narrow its own automatic PR triggers in its
323
+ Markdown frontmatter (alongside its other `config` settings):
324
+
325
+ ```yaml
326
+ config:
327
+ agents:
328
+ - agent: CODEX_CLI
329
+ checks:
330
+ paths: ["cli/**", "packages/shared/**"]
331
+ ```
332
+
333
+ Root `checks.paths`, when present, must match first. The App then runs matching
334
+ cases and all cases without a filter. Changes to `gauge.json` select every case;
335
+ editing a case selects that case regardless of its filter. If no cases match,
336
+ the check is skipped before preparing inputs or reserving credits. `config.inputs`
337
+ only chooses inputs; it does not define change triggers.
338
+
339
+ The first check uses the PR diff. After a passing evaluation, later updates
340
+ compare against that evaluated commit, so unrelated follow-up edits can skip.
341
+ A failed or unfinished evaluation, a changed target branch, or an unavailable
342
+ or incomplete diff causes a run. Skipped checks start no sessions and reserve
343
+ no credits. GitHub's rerun action runs regardless of paths; explicit CLI runs
344
+ also ignore this automatic-check filter.
345
+
346
+ File inputs stay outside the consumer repository and their paths are provided to
347
+ the agent. npm packages install with ordinary npm using the VM's Node/npm runtime.
348
+ `install` defaults to `environment`: a separate tools directory with the package's
349
+ own declared commands on PATH, including login shells. `workspace` adds a consumer
350
+ application dependency and accepts libraries without commands. Transitive registry
351
+ dependency ranges remain live. Plan output shows the installation destination and,
352
+ when a previous matching preparation captured it, command metadata. Otherwise it
353
+ reports that commands will be determined during preparation; planning never downloads
354
+ archives or starts VMs.
355
+
356
+ Native executables use `type: "binary"`, with already unpacked files in a GitHub
357
+ Actions artifact (or small committed Git files):
358
+
359
+ ```json
360
+ "gh": {
361
+ "type": "binary",
362
+ "source": {
363
+ "type": "github-actions",
364
+ "workflow": ".github/workflows/package.yml",
365
+ "artifact": "gh-linux"
366
+ },
367
+ "paths": ["dist/**"],
368
+ "platform": "linux-x64",
369
+ "commands": { "gh": "dist/bin/gh" }
370
+ }
371
+ ```
372
+
373
+ Paths are relative to the uploaded artifact root. Use your normal PR build;
374
+ for `actions/upload-artifact@v7`, use `archive: true`. Include companion files in
375
+ `paths` and explicitly map 1–32 command names to ordinary selected files. Gauge
376
+ validates Linux x86-64 ELF executables in its isolated artifact processor, restores
377
+ executable permissions, preserves the bundle outside the consumer repository, and
378
+ exposes the commands on PATH, including login shells. A candidate command overrides
379
+ an installed product command with the same name; two inputs selected by the same
380
+ case cannot expose the same command (including captured npm environment commands).
381
+
382
+ The first version supports ELF executables and PIE binaries compatible with the
383
+ consumer Linux runtime. It does not unpack nested release archives, install OS
384
+ packages, alter library search paths, or run arbitrary installation/version-check
385
+ commands. Prefer a static binary or build against a compatible Linux runtime;
386
+ header validation cannot prove shared-library compatibility. Scripts, macOS,
387
+ Windows, and ARM executables are unsupported. A missing loader/library encountered
388
+ when the agent invokes the command appears in session execution evidence, not in
389
+ static validation. Harness/runtime names (including shells, Node/npm, Python, Git,
390
+ archive/setup utilities, and agent commands) are reserved; use a different alias
391
+ for such a candidate. `GAUGE_INPUTS` gives the preserved bundle path.
392
+
393
+ A Skill input has `type: "skill"` and `path: "skills/product"`, or a path to a ZIP,
394
+ `.tar.gz`, or `.tgz` whose root contains `SKILL.md`. Use `path: "."` for a Skill at
395
+ the source root. Gauge validates and extracts selected archives in isolation and
396
+ stages the Skill in the coding agent's native discovery location. File inputs use
397
+ `paths` to select committed files or files inside an Actions artifact.
398
+
399
+ Gauge tests the artifact uploaded by your PR's successful build. Keep your
400
+ existing checkout settings. The GitHub check links to the selected build;
401
+ rerun Gauge to test the latest build.
402
+
403
+ ### Run from your own CI or local build
404
+
405
+ With CLI 0.18.0 or later, supply built inputs directly to run without the GitHub App:
406
+
407
+ ```sh
408
+ gauge --org my-org run --input product-analytics=./dist/skill.zip --yes -o json > request.json
409
+ gauge --org my-org run wait "$(jq -er '.runRequestId' request.json)" --check -o json > results.json
410
+ ```
411
+
412
+ `--check` fails the job unless the verdict is `PASSED` or `MEASURED`. Without
413
+ it, `run wait` exits 0 once the request settles, whatever the verdict.
414
+
415
+ Set `GAUGE_API_TOKEN` as a CI secret and replace `my-org` with your workspace slug.
416
+ This can run immediately after your build in the same CI job.
417
+
418
+ Repeat `--input name=path` for each input. Skills accept a directory containing
419
+ `SKILL.md` or a ZIP/tar.gz archive; npm inputs accept a packed `.tgz`. Files and
420
+ binary inputs accept a directory, with configured paths relative to that directory.
421
+ Use a dedicated build directory: links are rejected, and common credential files
422
+ and `node_modules` are excluded. Each input is limited to 512 MiB and 20,000 files.
423
+
424
+ With `--input`, Gauge uses your current evaluation files, including uncommitted
425
+ edits. No Git checkout is required when all selected inputs are supplied locally.
426
+ Inputs without an override keep their configured source. You can omit `source`
427
+ in `gauge.json` to default to Git; the override replaces it for this run.
428
+ Website previews and private consumer repositories still require their configured
429
+ access. Public consumer repositories and the default workspace need no GitHub App.
430
+
431
+ `wait -o json` returns the verdict, session results, criterion explanations, and
432
+ session links. It exits 0 once the request settles, whatever the verdict. Add
433
+ `--check` to exit 1 unless the verdict is `PASSED` or `MEASURED`; the results
434
+ JSON is still the only output. Automatic PR checks continue to use configured
435
+ sources; disable those checks in Gauge's GitHub settings if CI owns the trigger.
436
+ Save `results.json` as a CI artifact even when the check fails.
437
+
438
+ ### Authenticated MCP servers
439
+
440
+ Connect your account in Gauge's MCP setup, completing OAuth in your browser when
441
+ required. Add the server's versioned reference to each case's `config.mcpRefs`.
442
+ Gauge manages credentials for the session; keep tokens out of prompts and Git.
443
+
444
+ ### Website previews
445
+
446
+ Website inputs use `type: "website"` and a canonical HTTPS origin in `url`.
447
+ The preview can come from `source: { "type": "github-deployment", "environment": "preview" }`
448
+ or a `github-pr-comment` source with an exact `author`, literal `bodyIncludes`
449
+ marker, and a readiness `pattern` with named `url` and `sha` captures. Gauge selects
450
+ candidate-specific evidence; timestamps alone cannot establish the revision.
451
+ Comments must belong to a unique open same-repository PR at the candidate head.
452
+ Previews must be public HTTPS origins, without authentication or path prefixes.
453
+ They are fetched live rather than snapshotted.
454
+
455
+ Editor schemas are published at
456
+ `https://agents.withgauge.com/schemas/gauge/v2.7.0/gauge.json` and
457
+ `https://agents.withgauge.com/schemas/gauge/v2.7.0/case.json`. Associate them in editor
458
+ settings; `gauge.json` does not accept a `$schema` field. These release URLs are
459
+ immutable. Update earlier pilot configs by moving workflows into input sources,
460
+ replacing `use` with input `type` and `path`/`paths`, and moving cases into
461
+ `gauge-evals/` (or setting `cases` explicitly).
462
+
463
+ `--org` overrides the committed organization. Root `cases` can change discovery;
464
+ root `version` defaults to 2. Saved evals use `gauge run --eval <id>`.
465
+
466
+ ## Measurements own their configuration
467
+
468
+ Every runnable thing carries its own complete run configuration — repository,
469
+ persona, agent roster with model pins, skills, MCP servers, connections — plus
470
+ its own cadence and sample count. Nothing is shared, attached, or detached.
471
+
472
+ An **eval** is one prompt against one configuration:
473
+
474
+ ```sh
475
+ gauge suite create eval --name "add a healthcheck" \
476
+ --text "Add a /health endpoint that returns 200 OK." \
477
+ --criterion "Returns 200: the endpoint responds 200 with no auth" \
478
+ --repo https://github.com/acme/api --ref main \
479
+ --agent claude-code:claude-opus-4-8 --agent codex \
480
+ --skill acme-cli@1.2.0 \
481
+ --cadence weekly --samples 3
482
+ ```
483
+
484
+ `gauge suite edit eval <id>` changes any of it later; omitted flags keep their
485
+ stored value, and each `--clear-*` flag empties one field. To measure the same
486
+ prompt against a second repository or roster, copy the eval in the app and edit
487
+ the copy — one eval is always one configuration.
488
+
489
+ A **preference prompt** is a reusable question. It owns one set of run
490
+ settings — roster, resources, cadence, samples — and any number of
491
+ **scenarios**, each one repository × persona cell. Every scenario runs with
492
+ the prompt's settings, so one prompt measures the same question across several
493
+ repositories and personas at once:
494
+
495
+ ```sh
496
+ gauge suite create preference --name "CLI deployment preference" \
497
+ --kind head-to-head --brand-a acme --brand-b globex \
498
+ --text "Which CLI should I use to deploy?" \
499
+ --agent claude-code --cadence daily \
500
+ --repo https://github.com/acme/api # first scenario
501
+
502
+ gauge suite edit preference <promptId> --agent codex --samples 3 --cadence weekly
503
+ gauge suite scenarios list <promptId>
504
+ gauge suite scenarios add <promptId> --persona <profileId>
505
+ gauge suite scenarios add <promptId> --repo https://github.com/acme/web --persona <profileId>
506
+ gauge suite scenarios rm <promptId> --scenario <scenarioId>
507
+ ```
508
+
509
+ A prompt with zero scenarios is a valid definition — it simply never comes due.
510
+ Sessions per launch = scenarios × agent/model targets × samples. Omit the
511
+ run-settings flags on `create` and the org's default roster applies.
512
+
513
+ ## Cadence is the only off switch
514
+
515
+ `--cadence none` is "manual": the measurement stops coming due and runs only
516
+ when you launch it. There is no separate pause or enabled flag, in the CLI or
517
+ the app. `--cadence daily|weekly|monthly` puts it back on a rhythm, and
518
+ `suite get preference <id>` reads a manual prompt back as `cadence: manual`.
519
+
520
+ `gauge run --eval <id>` and `gauge run --preference <id>` are the separate
521
+ immediate launches. They use the stored configuration and ask before spending
522
+ (`--yes` skips the prompt for scripts).
523
+ `gauge run --preference <id> --scenario <scenarioId>` (repeatable) runs a subset;
524
+ omit it and every scenario runs. On a recurring cadence, an eval launch or a
525
+ full preference launch moves the next scheduled run to one period from now. A
526
+ scenario subset leaves the schedule unchanged.
527
+
528
+ ## Optimizations
529
+
530
+ Start by finding runs that encountered the surface you want to improve:
531
+
532
+ ```sh
533
+ gauge instructions optimization
534
+ gauge optimizations baselines --eval ev_1 -o json
535
+ gauge run insight run_1 -o json
536
+ gauge run get run_1 --logs -o json
537
+ gauge optimizations start --eval ev_1 --from-run run_1 --yes
538
+ gauge optimizations watch cl_1
539
+ ```
540
+
541
+ `baselines` returns observed owned surfaces grouped by run, source evidence,
542
+ judging status, eligibility (including the baseline's complete run set), and a
543
+ `nextCursor` for `--cursor`. It uses the UI baseline picker's compatibility rules
544
+ and recent 100-batch eligibility window. Missing evidence is unknown; the list
545
+ is an analysis summary, not an exhaustive inventory of every trace event.
546
+ `--from-run` inherits the eligible baseline's harness/model and reuses its runs.
547
+ A stale choice fails instead of silently buying a new baseline. `--target` is
548
+ an alternative when you need to establish a baseline; do not combine it with
549
+ `--from-run`. Use `--preference` instead of `--eval` for preference prompts.
550
+
551
+ ```sh
552
+ gauge optimizations authoring cl_1 > context.json
553
+ gauge optimizations page cl_1 --url https://docs.acme.dev/auth -o json
554
+ # Author round.json using context.revision and the observed baseline evidence.
555
+ gauge optimizations round preview cl_1 --file round.json -o json
556
+ gauge optimizations round save cl_1 --file round.json --digest <previewDigest>
557
+ gauge optimizations launch cl_1 --yes
558
+ gauge optimizations watch cl_1
559
+ gauge optimizations trial cl_1 R1.1 -o json
560
+ gauge optimizations adopt cl_1 R1.1 --yes
561
+ gauge optimizations handoff cl_1
562
+ gauge optimizations complete cl_1 --yes
563
+ ```
564
+
565
+ A round is `{revision, trials: [{trial: {title, hypothesis, lever, intervention}}]}`.
566
+ Docs `REWRITE_DOCS` proposals supply `after.pages` with URL and exact `edits`
567
+ (`find`/`replace`), or a complete `body` and `rewriteReason`. Check the page's
568
+ `source`: `scrape` means current fetched content, not the captured response.
569
+
570
+ For CLI trials, inspect `cliIntervention.guidance` and `targets` in authoring
571
+ and use `gauge run get <id> --logs` for the cited command evidence. `cli-change --file change.json`
572
+ accepts v3 `{schemaVersion:3, target, rules, shippingIntent}` and returns a sealed
573
+ bundle to put in `CLI_INTERVENTION.after`. It does not save or launch. The old
574
+ catalog/invocation-ID workflow is retired. Skills use `skill-files` and the
575
+ complete edited file bundle.
576
+
577
+ A checkpoint is a saved state of a baseline run. Resuming it avoids repeating
578
+ earlier work. Gauge automatically selects a compatible checkpoint before the
579
+ affected interaction for docs and CLI candidates, or starts fresh with a reason.
580
+ You do not select a checkpoint; omit checkpoint choices and fork flags. Skills
581
+ start fresh. Inspect
582
+ `runs[].candidateExecution` for actual starting mode and `candidateDelivery`
583
+ for exposure, observed artifacts, and evidence completeness. A score improvement
584
+ does not prove the change was exercised. Report sample counts and failed sessions.
585
+
586
+ Read `nextAction` on optimization detail; `watch` exits 4 when input is needed.
587
+ Round save replaces pending drafts but never launches. Use `--yes` for approved
588
+ spending and `trials add --idempotency-key` when retrying single-trial writes.
589
+ Adoption selects a winner without changing the baseline or deploying the change;
590
+ `handoff` supplies instructions to implement it in your source repository.
591
+
592
+ ## Compare runs
593
+
594
+ `gauge run compare` answers before/after questions in one call. Each side takes
595
+ repeatable `key=value` selectors: `request=`, `batch=`, `eval=`, `revision=`,
596
+ `experiment=`, `trial=`, `agent=`, `model=`, `since=`, and `until=`.
597
+
598
+ ```sh
599
+ gauge run compare --baseline request=<id> --candidate request=<id>
600
+ gauge run compare --baseline eval=<id> --baseline until=2026-10-01 \
601
+ --candidate eval=<id> --candidate since=2026-10-01
602
+ ```
603
+
604
+ It prints the pass rate, each criterion's pass rate, and median turns, cost,
605
+ and duration, for all runs and per agent and model. `-o json` returns the same
606
+ data with deltas.
607
+
608
+ With `--local` it compares runs in `.gauge/runs`. Local sides take `batch=`,
609
+ `eval=<case path>`, `agent=`, `model=`, `since=`, and `until=`. Local runs have
610
+ no judge verdicts, cost, or duration, so it compares completed runs and median
611
+ turns.
612
+
613
+ ```sh
614
+ gauge run compare --local --baseline batch=<id> --candidate batch=<id>
615
+ ```
616
+
617
+ Each run has an `evalRevision`: a hash of its prompt, repository, persona,
618
+ agent context, and criteria. `gauge suite edit` changes the revision of new
619
+ runs. Each case has its own revision. `gauge run list` shows it, and `compare`
620
+ warns when one eval or case ran under more than one revision; add
621
+ `revision=<hash>` to pick one. `compare` also warns when a side matched no
622
+ runs, has runs in progress, or has runs without a verdict.
623
+
624
+ Optimization trial runs test a changed configuration, so `eval=` and other
625
+ selectors leave them out. Add `trial=<trialId>` to compare one trial's runs,
626
+ including runs it reused. `gauge run list` shows the trial key and id.
627
+ `compare` prints the trial behind each side's trial runs. `gauge run list` and
628
+ `gauge run get` show a run's `optimizationTrial`.
629
+
630
+ To try other agents or sample counts once, without editing the suite, pass
631
+ `--agent` and `--samples` to `gauge run --eval <id>` or `--preference <id>`.
632
+ The stored configuration stays unchanged.
633
+
634
+ `gauge run status <request-id>` prints a launch request's progress and
635
+ per-session results once, without waiting.
636
+
637
+ ## Models, reports, and scripting
43
638
 
44
639
  Use `gauge models list` before pinning models. It reads the organization's live
45
640
  catalog and shows every selectable logical model with its harness, provider,
@@ -47,19 +642,107 @@ and effective capabilities. Open harnesses use `--agent pi:<model>` and
47
642
  `--agent opencode:<model>` just like the frontier harnesses.
48
643
 
49
644
  `gauge query` builds flexible reports over your event warehouse (a GA4
50
- `run_report` analog): pick a `--dataset`, group by `--dimensions` (including
51
- `run_preset`), and aggregate `--metrics`, or pass a whole spec with `--json`.
52
- `gauge query datasets` / `gauge query fields --dataset <name>` list what you
53
- can ask for.
54
-
55
- To launch runs declaratively, `gauge apply -f <spec>.json`. Discover the spec
56
- format without leaving the terminal: `gauge apply --example` prints a starter
57
- spec and `gauge apply --schema` prints its JSON Schema.
645
+ `run_report` analog): pick a `--dataset`, group by `--dimensions`, and aggregate
646
+ `--metrics`, or pass a whole spec with `--json`. `gauge query datasets` /
647
+ `gauge query fields --dataset <name>` list what you can ask for.
58
648
 
59
649
  Every read supports `-o json` for scripting. `GAUGE_API_TOKEN` overrides
60
650
  stored credentials (useful in CI). Exit codes: `0` ok · `1` error ·
61
- `2` usage · `3` billing-blocked.
651
+ `2` usage · `3` billing-blocked · `4` needs your input (`optimizations watch`).
62
652
 
63
653
  ## Support
64
654
 
65
655
  Questions or problems: [support@withgauge.com](mailto:support@withgauge.com).
656
+
657
+
658
+ ### Git-authored Agent Preference
659
+
660
+ A committed Markdown case may declare `preference` instead of `criteria`.
661
+ Use exactly one; the Markdown body is always the coding agent's user request.
662
+
663
+ ```markdown
664
+ ---
665
+ preference:
666
+ kind: open-ended
667
+ config:
668
+ agents:
669
+ - agent: CODEX_CLI
670
+ sampleCount: 5
671
+ inputs: [docs]
672
+ ---
673
+
674
+ I need product analytics for a small SaaS application.
675
+ We need funnels, retention analysis, and EU data residency.
676
+ What should we use?
677
+ ```
678
+
679
+ Here `docs` names an input declared in `gauge.json`. Existing repository/persona
680
+ settings and automatic path filters apply unchanged. Add `branded: true` when an
681
+ open-ended prompt names a brand. For a comparison, use:
682
+
683
+ ```yaml
684
+ preference:
685
+ kind: head-to-head
686
+ brandA: PostHog
687
+ brandB: Amplitude
688
+ ```
689
+
690
+ Write both brands in the prompt itself; metadata is only for analysis and never
691
+ rewrites the task. `gauge run --verify`, `--dry-run`, and `gauge run` support mixed experience
692
+ and preference suites. These fields require the CLI version containing this
693
+ feature; CLI 0.16.0 and earlier reject them.
694
+
695
+ Preference uses the existing app analyst: ranked picks for open-ended cases,
696
+ and a winner (or neither) for head-to-head. Results are advisory. Preference-only
697
+ GitHub checks finish neutral; experience criteria still gate mixed suites.
698
+ `gauge run wait --check` exits successfully for `MEASURED` or `PASSED`;
699
+ input, execution, and analysis errors exit 1. Results describe the
700
+ candidate revision and do not infer a base/head delta.
701
+
702
+ ## Local process lifecycle
703
+
704
+ `gauge run --local` runs in the foreground. Ctrl+C (or SIGTERM) stops the
705
+ queue, cancels active sessions, waits for cleanup, and exits with code 130.
706
+ Partial results and traces remain under `.gauge/runs`. Successful sessions
707
+ remove their containers and temporary credentials; the Docker image stays cached.
708
+
709
+ Use `gauge run --detach --local` to keep a batch running after the CLI exits.
710
+
711
+ Use `gauge run --local --runtime-image my-runtime:dev` to select a compatible
712
+ custom image, including with `--dry-run` or `--detach`. Derive from the standard
713
+ runtime to add Python or other system dependencies; see the
714
+ [Python example](https://github.com/gauge-sh/gauge-cli/tree/main/examples/runtime-python).
715
+ Preserve the runtime's tools, contract label, PATH, host-user permissions, and
716
+ entrypoint behavior. Gauge validates
717
+ the image before accepting a batch and pins its immutable ID for queued sessions.
718
+ A detached supervisor drains the queue and exits when the batch settles. The
719
+ launch prints its batch ID and these commands, to run from the same repository:
720
+
721
+ ```sh
722
+ gauge run status --batch <batch-id> --local
723
+ gauge run wait --batch <batch-id> --local
724
+ gauge run cancel --batch <batch-id> --local
725
+ ```
726
+
727
+ `gauge run wait --batch <batch-id> --local` blocks until the batch settles
728
+ (`completed`, `canceled`, `failed`, or `interrupted`), then prints each
729
+ session's state and the next step. It exits 0 once settled; `--check` exits 1
730
+ unless every session completed.
731
+
732
+ `gauge run status <run-id> --local` inspects a single session;
733
+ `gauge run cancel <run-id> --local` cancels only that run. All support `-o json`.
734
+ Batch cancellation is persisted before workers are contacted, including during
735
+ startup. A pending cancellation returns nonzero; inspect status before retrying.
736
+
737
+ After an interruption, `gauge cleanup --local` retries cleanup for recorded
738
+ interrupted sessions or sessions with incomplete cleanup in this repository.
739
+ It preserves evidence and recovered work, and makes recovered results visible
740
+ in `gauge run list`. It leaves live sessions, other repositories, unrelated
741
+ containers, and cached images alone. Recovery checks Gauge's container label
742
+ and workspace mount before removing a remaining container. Uncertain worker
743
+ ownership is reported rather than signaling a potentially reused PID.
744
+
745
+ Local workers have a 30-minute turn deadline and a 60-second supervisor
746
+ response deadline, so losing a supervisor cannot leave an idle worker forever.
747
+ Cleanup cannot discover sessions whose `.gauge` records were deleted; keep
748
+ those records until cleanup has succeeded.