@withgauge/cli 0.21.0 → 0.23.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -7,31 +7,220 @@ live from your terminal.
7
7
  ```sh
8
8
  npm install -g @withgauge/cli
9
9
  gauge update # update the CLI to the latest published version
10
- gauge onboard # sign in, create a workspace, seed starter measurements
10
+ gauge onboard # sign in and create a workspace (account only)
11
11
  gauge auth login # (or just sign in)
12
12
  gauge orgs use <org> # set your default organization
13
+ gauge init # initialize a workspace; local is the default
14
+ gauge init --hosted # set a private hosted backend preference
15
+ gauge init --cases # optionally scaffold shared gauge-cases/
13
16
  ```
14
17
 
15
- Requires Node.js ≥ 22.12.
18
+ Requires Node.js ≥ 22.13.
16
19
 
17
- To install the optional
18
- [`gauge-agents` skill](https://github.com/gauge-sh/gauge-skills/tree/main/gauge-agents)
19
- globally, run:
20
+ ## Workspace and local storage
21
+
22
+ `gauge init --local` works without Git. It writes a minimal shared manifest:
23
+
24
+ ```json
25
+ { "version": 2, "inputs": {} }
26
+ ```
27
+
28
+ The private `.gauge/config.json` contains `{ "version": 1, "backend": "local" }`.
29
+ `.gauge/.gitignore` ignores all workspace state. SQLite in `.gauge/gauge.sqlite`
30
+ stores saved measurements, revisions, launch/session summaries, and published
31
+ analyses. Raw transcripts, normalized events, diffs, and diagnostics remain under
32
+ `.gauge/runs/`; input blobs and batch lifecycle files remain alongside them.
33
+
34
+ Moving the workspace preserves identity. Fresh clones or linked worktrees start
35
+ independently; switching Git branches keeps local history. Do not treat `.gauge/`
36
+ as a disposable cache. Credentials and user preferences stay user-level.
37
+ `gauge run repair --local` rebuilds session summaries from durable artifacts while
38
+ preserving authored measurements and analyses. It does not rerun sessions or judges.
39
+
40
+ ## Analyze local sessions
41
+
42
+ Your coding agent reads evidence and publishes a structured analysis:
43
+
44
+ ```sh
45
+ gauge instructions session-analysis --local
46
+ gauge run analysis schema --local > analysis.schema.json
47
+ gauge run get <run-id> --logs --local -o json
48
+ # Author analysis.json against the printed schema and session evidence.
49
+ gauge run analysis save <run-id> --file analysis.json --local
50
+ gauge run insight <run-id> --local
51
+ ```
52
+
53
+ Publication validates the versioned schema, criterion coverage, and evidence
54
+ references, then saves findings, product choices, and criterion judgments in one
55
+ transaction. Reanalysis creates a revision and updates the current analysis.
56
+ `run insight` leads with Friction, Objections, and Opportunities in either backend.
57
+ A Markdown file alone does not mark a session analyzed. Unknown future schema
58
+ versions produce an upgrade error without altering saved data. Database schema
59
+ migrations preserve analyses and never silently reinterpret old judgments.
60
+
61
+ ## Local Docker runtime
62
+
63
+ Local runs need Docker running Linux containers and an agent credential. Linux
64
+ is the container OS, not a requirement for your host: Docker Desktop on macOS
65
+ provides a Linux daemon and shares the local files used by sessions.
66
+ Install the runtime without a Gauge account or a source checkout:
67
+
68
+ ```sh
69
+ gauge local runtime pull
70
+ gauge local runtime inspect -o json
71
+ gauge run --local --dry-run
72
+ ```
73
+
74
+ Each CLI release pins its tested public `ghcr.io/gauge-sh/gauge-runtime` image by
75
+ immutable digest. Inspection reports the reference, local image ID, registry
76
+ digest, contract, platform, and readiness errors. `ready` covers local image
77
+ metadata; agent credential checks happen separately. Neither inspection nor a
78
+ dry run downloads an image. `pull` explicitly downloads the supported image;
79
+ previously pulled images can be reused offline (agent calls still need network
80
+ access). Pull progress goes to stderr so JSON output remains parseable.
81
+
82
+ The runtime includes native `linux/amd64` and `linux/arm64` images. Docker selects
83
+ the matching image on Intel/AMD or Apple Silicon/ARM machines. No architecture
84
+ emulation is needed. Docker must be able to bind-mount this machine's temporary
85
+ and input directories; an unrelated remote Docker server cannot access them.
86
+ On Windows, run the CLI inside WSL because Gauge uses Unix sockets and UID/GID.
87
+
88
+ ## Local agent credentials
89
+
90
+ Local provider credentials are separate from a Gauge account and hosted credential
91
+ brokering. Use native harness logins or configure a named environment reference:
92
+
93
+ ```sh
94
+ gauge local profiles set work --agent CODEX_CLI --provider openai --auth api_key --env TEAM_OPENAI_KEY
95
+ gauge local profiles use work
96
+ gauge local profiles inspect work -o json
97
+ gauge run --local --profile work --dry-run -o json
98
+ ```
99
+
100
+ `--env` takes the variable's name, never its secret value. Profiles store only
101
+ references; list and inspect output do not expose credentials. Update a profile
102
+ with `set ... --replace` or remove it with `remove <profile>` without deleting
103
+ the native login. Local readiness checks do not verify provider acceptance or
104
+ model access. Use `gauge local profiles --help` for profile commands and
105
+ `gauge instructions --local` for the local workflow.
106
+
107
+ ## Anonymous CLI telemetry
108
+
109
+ The installed CLI collects command usage by default to help improve Gauge.
110
+ Opt out for your user account or for a single process:
20
111
 
21
112
  ```sh
22
- npx --yes skills@1.5.0 add gauge-sh/gauge-skills --skill gauge-agents --global --yes
113
+ gauge telemetry disable
114
+ gauge telemetry status
115
+ gauge telemetry enable
116
+ GAUGE_TELEMETRY_DISABLED=1 gauge run --verify
23
117
  ```
24
118
 
119
+ The environment variable overrides the saved preference. These controls work
120
+ without an account or network access and support `-o json`. They apply to CLI
121
+ command telemetry; authenticated Gauge API requests have separate server-side
122
+ usage analytics.
123
+
124
+ Each `cli_command_invoked` event contains only the canonical command name
125
+ (for example, `run list`), its UTC invocation timestamp, CLI version, and a
126
+ random installation UUID. Arguments, flag values, file paths, prompts, file
127
+ contents, environment variables, errors, and account identifiers are excluded.
128
+ The UUID is independent of login and is never linked to your Gauge account.
129
+ This is pseudonymous usage tracking: an installation can be recognized across
130
+ commands, but no personal identity is sent. HTTP requests necessarily expose a
131
+ source IP to the receiving infrastructure; event properties mask IP addresses,
132
+ disable GeoIP enrichment, and disable person-profile creation.
133
+
134
+ Preferences live in `~/.config/gauge/telemetry.json` and the UUID in
135
+ `~/.config/gauge/telemetry-id` (respecting `XDG_CONFIG_HOME`). They are shared
136
+ across repositories, independently of each repository's `.gauge/` results.
137
+ Deleting `telemetry-id` resets the identity on the next collected command.
138
+ Consumers are installations, not necessarily individual people. Session
139
+ analysis can group an installation's events using a 30-minute inactivity gap;
140
+ the CLI does not generate a different identity for each invocation.
141
+
142
+ Collection occurs when a parsed command is about to run, including commands
143
+ whose handlers subsequently fail. Help, version, parsing errors, telemetry
144
+ controls, and source development runs are excluded. A detached worker handles
145
+ all telemetry storage and network access, with a two-second request timeout
146
+ and a three-second worker lifetime. Commands never wait for delivery, and
147
+ failures are silent. There are no retries or offline queue, so events may be
148
+ lost. Corrupt or unreadable preferences suppress collection until repaired.
149
+
150
+ Events go to Gauge's existing PostHog project using its public ingestion key.
151
+ The [capture API](https://posthog.com/docs/api/capture) and
152
+ [anonymous event controls](https://posthog.com/docs/product-analytics/capture-events#advanced-anonymous-vs-identified-events)
153
+ describe the transport and person-profile behavior.
154
+
155
+ ## Local or hosted
156
+
157
+ For an end-to-end improvement loop, run `gauge instructions hillclimb`. It guides
158
+ your coding agent to identify product inputs, propose a representative pilot,
159
+ inspect and publish findings, implement a focused change, and rerun to compare.
160
+ Init and the generated `AGENTS.md` point to this guidance. `gauge install-skill`
161
+ installs the Gauge skill, which directs agents to the installed CLI's instructions.
162
+
163
+ Execution backend is independent of definition source. Resolve an explicit
164
+ `--local`/`--hosted` flag, then `.gauge/config.json`, user preference, and finally
165
+ local. Signing in does not change the backend. `gauge.json` is the shared project
166
+ manifest for named inputs, optional case discovery, and GitHub checks. It never
167
+ contains the execution preference. Git is optional for workspace initialization.
168
+
169
+ | Workflow | Local | Hosted |
170
+ |---|---|---|
171
+ | Skill, npm package (CLI or SDK), and `files` inputs | yes | yes |
172
+ | Docs supplied as `files` inputs | yes | yes |
173
+ | Eval and Agent Preference cases in `gauge-cases/` | yes (you read the sessions) | yes (judged) |
174
+ | `run`, `run list`, `run compare --local`, `models list`, `cleanup`, `feedback` | yes | yes |
175
+ | Binary, website, and MCP inputs | no | yes |
176
+ | Saved eval sets and preference prompts | yes | yes |
177
+ | Project inputs and previews on saved measurement launches | yes | use case files/specs |
178
+ | Findings and summary queries | yes, supported fields | yes |
179
+ | Personas, recurring schedules, PR checks | no | yes |
180
+ | Optimizations (managed docs rewriting) | no | yes |
181
+ | Stored credentials for MCP servers and connections | no | yes |
182
+ | Additional reporting datasets, dashboards, org settings | no | yes |
183
+
184
+ Local runs need Docker and the agent's own login (`claude` or
185
+ `ANTHROPIC_API_KEY` for Claude Code; `codex login` for Codex, Pi, and
186
+ OpenCode). They do not need a Gauge account. Hosted runs need a Gauge account
187
+ (`gauge onboard`) and use credits. In local mode, a hosted-only command says so
188
+ and prints the next step.
189
+
190
+ For coding agents, we recommend the
191
+ [`gauge-agents` skill](https://github.com/gauge-sh/gauge-skills/tree/main/gauge-agents).
192
+ It makes Gauge CLI guidance and repository setup workflows discoverable in future
193
+ sessions. Install it separately from repo setup, running from your project root:
194
+
195
+ ```sh
196
+ gauge install-skill
197
+ ```
198
+
199
+ In a terminal, select your coding agent and installation scope in the installer.
200
+ Project scope adds skill files to this repo; global scope makes the skill available
201
+ across projects. The command runs the pinned `skills@1.5.0` installer through npx
202
+ and forwards its prompts and output. When a coding agent runs it, the command
203
+ installs for that agent at project scope without prompts and prints a short summary.
204
+ To choose the agent or scope explicitly:
205
+
206
+ ```sh
207
+ gauge install-skill --agent claude-code --yes # this project
208
+ gauge install-skill --agent codex --global --yes # across projects
209
+ ```
210
+
211
+ Repeat `--agent` to target multiple agents. `--yes` requires an explicit agent
212
+ target; `--copy` copies files instead of creating symlinks.
213
+
25
214
  ## Usage
26
215
 
27
216
  ```sh
28
217
  gauge status # the org at a glance
29
- gauge evals list # judged agent-experience measurements
30
- gauge preference list # Agent Preference prompts and measurements
218
+ gauge eval list # saved eval sets
219
+ gauge preference list # saved Agent Preference prompts
31
220
  gauge models list # supported model × harness × provider targets
32
- gauge evals edit <id> --cadence weekly --samples 3 # when and how much it runs
33
- gauge evals run <id> # launch now (ORG-funded, credit-gated)
34
- gauge runs watch <run-id> # follow a run as the agent works
221
+ gauge eval edit <id> --cadence weekly --samples 3 # when and how much it runs
222
+ gauge run --eval <id> # launch now (ORG-funded, credit-gated)
223
+ gauge run watch <run-id> # follow a run as the agent works
35
224
  gauge stats rankings -o json # brand rankings (share of voice) as JSON
36
225
  gauge query -d ecosystem -m installs,install_rate -o json # a flexible report
37
226
  ```
@@ -39,27 +228,28 @@ gauge query -d ecosystem -m installs,install_rate -o json # a flexible report
39
228
  `gauge instructions` prints a Gauge overview, CLI setup and usage rules, and
40
229
  roughly 100 words of essential guidance per module. Each summary points to
41
230
  `gauge instructions <topic>` for the full page. `gauge --help` points agents at
42
- it, or write the file into the repo with `gauge instructions --install`.
43
- The text ships inside the CLI, so it
44
- never drifts from the commands it describes.
45
-
46
- Use `gauge instructions evals` before creating evals: it covers developer tasks,
47
- model selection, observable judging criteria, and a complete migration example.
48
- Use `gauge instructions preference` to investigate product choices, source
49
- exposure, and interventions, from individual traces to larger cohorts.
231
+ it. Install the Gauge skill with `gauge install-skill` to make this guidance
232
+ discoverable in future sessions. Read instructions from the installed CLI so
233
+ they match its commands.
234
+
235
+ Use `gauge instructions run` before writing evals or starting runs: it covers
236
+ developer tasks, model selection, observable judging criteria, and a complete
237
+ migration example. Use `gauge instructions eval` and `gauge instructions preference` to save eval sets and
238
+ preference prompts, and `gauge instructions session-analysis` to investigate
239
+ product choices, source exposure, and interventions, from individual traces to
240
+ larger cohorts.
50
241
  Use `gauge instructions optimization` for the full optimization workflow.
51
242
  `gauge instructions --list` shows available topics.
52
- `gauge instructions evals --install` installs the eval guidance as a local skill.
53
243
 
54
244
  Run `gauge --help` (or `gauge <command> --help`) for the full command set:
55
- evals, preference, models, personas, skills, mcp, repos, runs, optimizations,
56
- batches, brands, tags, actions, dashboards, members, org, provider keys,
245
+ run, eval, preference, models, personas, skills, mcp, repos, optimizations,
246
+ brands, tags, actions, dashboards, members, org, provider keys,
57
247
  billing, usage, stats, query, and more.
58
248
 
59
249
  ## Run committed Markdown evals
60
250
 
61
251
  File evals keep the task, run settings, and judged criteria in Git. Create
62
- `gauge-evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
252
+ `gauge-cases/healthcheck.md` with YAML frontmatter and a Markdown task body:
63
253
 
64
254
  ```markdown
65
255
  ---
@@ -76,19 +266,28 @@ config:
76
266
  Add a /health endpoint and run the relevant app checks.
77
267
  ```
78
268
 
79
- Check your setup, then commit the files before planning and running with Git inputs:
269
+ Check your setup, then preview and run the current working-tree cases and inputs:
80
270
 
81
271
  ```sh
82
- gauge evals verify # offline; includes uncommitted working-tree edits
83
- gauge evals verify -o json # versioned global/per-input yes/no/unknown checklist
84
- gauge evals plan
85
- gauge evals run --yes
86
- gauge run-requests wait <request-id> -o json
272
+ gauge run --verify # local; includes uncommitted working-tree edits
273
+ gauge run --verify -o json # versioned global/per-input yes/no/unknown checklist
274
+ gauge run --dry-run
275
+ gauge run --yes
276
+ gauge run wait <request-id> -o json
87
277
  ```
88
278
 
89
- Cases default to `gauge-evals/**/*.md`. `--files` accepts quoted paths/globs and overrides
90
- that convention for `plan` and `run`. By default, these commands read committed Git
91
- HEAD. With `--input`, they read your current files, including uncommitted edits.
279
+ Cases default to `gauge-cases/**/*.md`. `--files` accepts quoted paths/globs and overrides
280
+ that convention for `--dry-run` and `run`. Interactive CLI launches snapshot current
281
+ working-tree definitions in either backend, including uncommitted edits. Selected
282
+ filesystem inputs are packaged for local execution or hosted upload; explicit remote
283
+ artifacts retain their declared source. GitHub checks fetch committed definitions.
284
+ Content digests identify the exact bytes separately from Git provenance.
285
+ Preview `inputSnapshots` includes the selected source `path`,
286
+ `gaugeArchive: { sha256, bytes }`, and the archived `files` (the first 100) with
287
+ `fileCount`. Those values describe Gauge's delivery archive, which contains the
288
+ selected files. `unattachedInputs` names the `gauge.json` inputs that the launch
289
+ leaves out. For npm inputs it contains the `.tgz`; the raw
290
+ tarball has a different hash and size. Each launch takes a fresh snapshot.
92
291
  Each case owns its agents, models, consumer repository/ref, Persona and criteria.
93
292
  The optional `config.inputs` selects named inputs; all attach by default.
94
293
 
@@ -107,7 +306,7 @@ account settings are outside this command's scope and do not create unknown rows
107
306
  Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
108
307
  No workflow code is executed.
109
308
 
110
- `gauge evals verify -o json` writes exactly one JSON object to stdout, including on
309
+ `gauge run --verify -o json` writes exactly one JSON object to stdout, including on
111
310
  validation failure. The versioned report provides:
112
311
 
113
312
  - `version: 1` and `mode: "local-static"` identify the report contract and scope.
@@ -125,7 +324,7 @@ validation failure. The versioned report provides:
125
324
  For example, an agent can select unresolved checks with:
126
325
 
127
326
  ```sh
128
- gauge evals verify -o json | jq '{status, facts, global: [.global[] | select(.state != "yes")], inputs: [.inputs[] | {name, checks: [.checks[] | select(.state != "yes")]}]}'
327
+ gauge run --verify -o json | jq '{status, facts, global: [.global[] | select(.state != "yes")], inputs: [.inputs[] | {name, checks: [.checks[] | select(.state != "yes")]}]}'
129
328
  ```
130
329
 
131
330
  An optional `gauge.json` declares inputs produced by your normal CI or committed
@@ -271,16 +470,24 @@ rerun Gauge to test the latest build.
271
470
  With CLI 0.18.0 or later, supply built inputs directly to run without the GitHub App:
272
471
 
273
472
  ```sh
274
- gauge --org my-org evals run --input product-analytics=./dist/skill.zip --yes -o json > request.json
275
- gauge --org my-org run-requests wait "$(jq -er '.runRequestId' request.json)" -o json > results.json
473
+ gauge --org my-org run --input product-analytics=./dist/skill.zip --yes -o json > request.json
474
+ gauge --org my-org run wait "$(jq -er '.runRequestId' request.json)" --check -o json > results.json
276
475
  ```
277
476
 
477
+ `--check` fails the job unless the verdict is `PASSED` or `MEASURED`. Without
478
+ it, `run wait` exits 0 once the request settles, whatever the verdict.
479
+
278
480
  Set `GAUGE_API_TOKEN` as a CI secret and replace `my-org` with your workspace slug.
279
481
  This can run immediately after your build in the same CI job.
280
482
 
281
- Repeat `--input name=path` for each input. Skills accept a directory containing
282
- `SKILL.md` or a ZIP/tar.gz archive; npm inputs accept a packed `.tgz`. Files and
283
- binary inputs accept a directory, with configured paths relative to that directory.
483
+ Repeat `--input name=path` for each input. The path replaces the input's
484
+ configured files entirely; only its type and settings carry over. It may be a
485
+ glob, using the same syntax as `gauge.json`; quote it so the shell passes it
486
+ through. Skills accept a directory containing `SKILL.md` or a ZIP/tar.gz archive;
487
+ npm inputs accept a packed `.tgz` or a glob matching exactly one. Files inputs
488
+ accept a directory, a single file, or a glob; archive paths are relative to the
489
+ directory, or to the glob's directory before its first wildcard. Binary inputs
490
+ accept a directory or glob, with `commands` relative to it.
284
491
  Use a dedicated build directory: links are rejected, and common credential files
285
492
  and `node_modules` are excluded. Each input is limited to 512 MiB and 20,000 files.
286
493
 
@@ -292,11 +499,32 @@ Website previews and private consumer repositories still require their configure
292
499
  access. Public consumer repositories and the default workspace need no GitHub App.
293
500
 
294
501
  `wait -o json` returns the verdict, session results, criterion explanations, and
295
- session links. It exits nonzero for failed evaluations or execution errors and
296
- still writes the results JSON. Automatic PR checks continue to use configured
502
+ session links. It exits 0 once the request settles, whatever the verdict. Add
503
+ `--check` to exit 1 unless the verdict is `PASSED` or `MEASURED`; the results
504
+ JSON is still the only output. Automatic PR checks continue to use configured
297
505
  sources; disable those checks in Gauge's GitHub settings if CI owns the trigger.
298
506
  Save `results.json` as a CI artifact even when the check fails.
299
507
 
508
+ ### Antigravity (hosted)
509
+
510
+ Antigravity is available to organizations enrolled in Antigravity access. Run
511
+ `gauge models list --hosted` to find the available model identifiers, then select
512
+ one with `--agent antigravity:<model>` when creating or launching a saved eval or
513
+ preference prompt. Case files and JSON specs use `agent: ANTIGRAVITY` with the
514
+ model identifier in `models`.
515
+
516
+ Launch with `--hosted`, for example:
517
+
518
+ ```sh
519
+ gauge run --eval <id> --agent antigravity:<model> --hosted
520
+ ```
521
+
522
+ For case files or a JSON case spec, preview with `gauge run --hosted --dry-run`
523
+ or `gauge run --apply cases.json --hosted --dry-run` before launching.
524
+ Hosted previews contact Gauge and may upload inputs; launches use credits.
525
+ The server checks organization access and model availability. Antigravity cannot
526
+ run locally yet.
527
+
300
528
  ### Authenticated MCP servers
301
529
 
302
530
  Connect your account in Gauge's MCP setup, completing OAuth in your browser when
@@ -312,31 +540,43 @@ marker, and a readiness `pattern` with named `url` and `sha` captures. Gauge sel
312
540
  candidate-specific evidence; timestamps alone cannot establish the revision.
313
541
  Comments must belong to a unique open same-repository PR at the candidate head.
314
542
  Previews must be public HTTPS origins, without authentication or path prefixes.
315
- They are fetched live rather than snapshotted. See the [preview source examples and
316
- limitations](../specs/docs-web-inputs.md), including Astro's existing bot comment.
543
+ They are fetched live rather than snapshotted.
544
+
545
+ Canonical-site preview routing requires hosted execution. Local runs reject
546
+ website inputs, including `--input` overrides, before checking Docker or credentials.
547
+ For local content-only testing, supply docs as a `files` input or put a reachable
548
+ preview URL directly in the prompt. Inputs never include `.gauge/`. Confirm that the preview contains the edits
549
+ and is reachable from the agent's environment; the host's localhost is not
550
+ automatically reachable from a container. These alternatives change exposure and
551
+ do not test discovery of the canonical docs. Static verification validates
552
+ configuration; it does not prove backend support, delivery, or an eval pass.
317
553
 
318
554
  Editor schemas are published at
319
- `https://agents.withgauge.com/schemas/gauge/v2.7.0/gauge.json` and
320
- `https://agents.withgauge.com/schemas/gauge/v2.7.0/case.json`. Associate them in editor
555
+ `https://agents.withgauge.com/schemas/gauge/v2.8.0/gauge.json` and
556
+ `https://agents.withgauge.com/schemas/gauge/v2.8.0/case.json`. Associate them in editor
321
557
  settings; `gauge.json` does not accept a `$schema` field. These release URLs are
322
- immutable. Update earlier pilot configs by moving workflows into input sources,
323
- replacing `use` with input `type` and `path`/`paths`, and moving cases into
324
- `gauge-evals/` (or setting `cases` explicitly).
558
+ immutable. Cases default to `gauge-cases/`, or use explicit `cases` paths.
325
559
 
326
560
  `--org` overrides the committed organization. Root `cases` can change discovery;
327
- root `version` defaults to 2. Saved evals use `gauge evals run <id>`. Superusers
328
- use platform funding by default and can add `--bill-to-org`.
561
+ root `version` defaults to 2. Saved evals use `gauge run --eval <id>`.
329
562
 
330
563
  ## Measurements own their configuration
331
564
 
332
- Every runnable thing carries its own complete run configuration — repository,
333
- persona, agent roster with model pins, skills, MCP servers, connections — plus
334
- its own cadence and sample count. Nothing is shared, attached, or detached.
565
+ Saved eval sets and preference prompts provide a stable grouping for sessions
566
+ plus editable templates and defaults. Local records live in workspace SQLite;
567
+ hosted records live in your organization. Launches freeze identity, revision, and
568
+ effective settings. Renaming or editing a measurement preserves its history.
569
+ Local measurements run on demand; recurring schedules, personas, markets, and
570
+ hosted resource references require hosted execution. Measurements do not store
571
+ project inputs: attach them to a local launch with
572
+ `gauge run --eval <id> --input <name>`, or `--input <name>=<path>` to replace the
573
+ configured files.
335
574
 
336
- An **eval** is one prompt against one configuration:
575
+ An **eval** is one prompt against one configuration. This hosted example uses
576
+ registered skills and recurring scheduling; omit those hosted features locally:
337
577
 
338
578
  ```sh
339
- gauge evals create --name "add a healthcheck" \
579
+ gauge eval create --name "add a healthcheck" \
340
580
  --text "Add a /health endpoint that returns 200 OK." \
341
581
  --criterion "Returns 200: the endpoint responds 200 with no auth" \
342
582
  --repo https://github.com/acme/api --ref main \
@@ -345,12 +585,14 @@ gauge evals create --name "add a healthcheck" \
345
585
  --cadence weekly --samples 3
346
586
  ```
347
587
 
348
- `gauge evals edit <id>` changes any of it later; omitted flags keep their
588
+ `gauge eval edit <id>` changes any of it later; omitted flags keep their
349
589
  stored value, and each `--clear-*` flag empties one field. To measure the same
350
590
  prompt against a second repository or roster, copy the eval in the app and edit
351
591
  the copy — one eval is always one configuration.
352
592
 
353
- A **preference prompt** is a reusable question. It owns one set of run
593
+ A **preference prompt** is a reusable question. The following hosted example
594
+ uses scenarios and recurring scheduling. Locally, save a prompt and launch it
595
+ on demand with `gauge run --preference <id> --local`. It owns one set of run
354
596
  settings — roster, resources, cadence, samples — and any number of
355
597
  **scenarios**, each one repository × persona cell. Every scenario runs with
356
598
  the prompt's settings, so one prompt measures the same question across several
@@ -374,32 +616,30 @@ A prompt with zero scenarios is a valid definition — it simply never comes due
374
616
  Sessions per launch = scenarios × agent/model targets × samples. Omit the
375
617
  run-settings flags on `create` and the org's default roster applies.
376
618
 
377
- ## Cadence is the only off switch
619
+ ## Hosted schedules
378
620
 
379
621
  `--cadence none` is "manual": the measurement stops coming due and runs only
380
622
  when you launch it. There is no separate pause or enabled flag, in the CLI or
381
623
  the app. `--cadence daily|weekly|monthly` puts it back on a rhythm, and
382
- `preference get` reads a manual prompt back as `cadence: manual`.
624
+ `gauge preference get <id>` reads a manual prompt back as `cadence: manual`.
383
625
 
384
- `gauge evals run <id>` and `gauge preference run <id>` are the separate
626
+ `gauge run --eval <id>` and `gauge run --preference <id>` are the separate
385
627
  immediate launches. They use the stored configuration and ask before spending
386
628
  (`--yes` skips the prompt for scripts).
387
- `gauge preference run <id> --scenario <scenarioId>` (repeatable) runs a subset;
388
- omit it and every scenario runs. Neither launch touches the cadence clock.
389
- Superusers can use `--org <slug>` to launch in any org. Their runs use PLATFORM
390
- funding by default, like Run Now in the web UI; `--bill-to-org` uses that org's
391
- credits instead. Other users always use org credits and can launch only in orgs
392
- they belong to.
629
+ `gauge run --preference <id> --scenario <scenarioId>` (repeatable) runs a subset;
630
+ omit it and every scenario runs. On a recurring cadence, an eval launch or a
631
+ full preference launch moves the next scheduled run to one period from now. A
632
+ scenario subset leaves the schedule unchanged.
393
633
 
394
- ## Optimizations
634
+ ## Optimizations (hosted)
395
635
 
396
636
  Start by finding runs that encountered the surface you want to improve:
397
637
 
398
638
  ```sh
399
639
  gauge instructions optimization
400
640
  gauge optimizations baselines --eval ev_1 -o json
401
- gauge runs insight run_1 -o json
402
- gauge runs logs run_1 -o json
641
+ gauge run insight run_1 -o json
642
+ gauge run get run_1 --logs -o json
403
643
  gauge optimizations start --eval ev_1 --from-run run_1 --yes
404
644
  gauge optimizations watch cl_1
405
645
  ```
@@ -434,7 +674,7 @@ Docs `REWRITE_DOCS` proposals supply `after.pages` with URL and exact `edits`
434
674
  `source`: `scrape` means current fetched content, not the captured response.
435
675
 
436
676
  For CLI trials, inspect `cliIntervention.guidance` and `targets` in authoring
437
- and use `runs logs` for the cited command evidence. `cli-change --file change.json`
677
+ and use `gauge run get <id> --logs` for the cited command evidence. `cli-change --file change.json`
438
678
  accepts v3 `{schemaVersion:3, target, rules, shippingIntent}` and returns a sealed
439
679
  bundle to put in `CLI_INTERVENTION.after`. It does not save or launch. The old
440
680
  catalog/invocation-ID workflow is retired. Skills use `skill-files` and the
@@ -455,6 +695,53 @@ spending and `trials add --idempotency-key` when retrying single-trial writes.
455
695
  Adoption selects a winner without changing the baseline or deploying the change;
456
696
  `handoff` supplies instructions to implement it in your source repository.
457
697
 
698
+ ## Compare runs
699
+
700
+ `gauge run compare` answers before/after questions in one call. Each side takes
701
+ repeatable `key=value` selectors: `request=`, `batch=`, `eval=`, `revision=`,
702
+ `experiment=`, `trial=`, `agent=`, `model=`, `since=`, and `until=`.
703
+
704
+ ```sh
705
+ gauge run compare --baseline request=<id> --candidate request=<id>
706
+ gauge run compare --baseline eval=<id> --baseline until=2026-10-01 \
707
+ --candidate eval=<id> --candidate since=2026-10-01
708
+ ```
709
+
710
+ It prints the pass rate, each criterion's pass rate, and median turns, cost,
711
+ and duration, for all runs and per agent and model. `-o json` returns the same
712
+ data with deltas.
713
+
714
+ With `--local` it compares local session summaries. Use saved measurement IDs
715
+ with `eval=` and `revision=<number>`, or select a `batch`, `agent`, `model`, and
716
+ time window. Compare preference runs by batch. The comparison uses current published
717
+ criterion judgments and recorded turns/duration; unknown metrics and unassessable
718
+ criteria stay out of their denominators. Completion is distinct from a judged result.
719
+ Inspect findings with `gauge run insight` and query capabilities with `gauge query fields`.
720
+
721
+ ```sh
722
+ gauge run compare --local --baseline batch=<id> --candidate batch=<id>
723
+ ```
724
+
725
+ Each run has an `evalRevision`: a hash of its prompt, repository, persona,
726
+ agent context, and criteria. `gauge eval edit` changes the revision of new
727
+ runs. Each case has its own revision. `gauge run list` shows it, and `compare`
728
+ warns when one eval or case ran under more than one revision; add
729
+ `revision=<hash>` to pick one. `compare` also warns when a side matched no
730
+ runs, has runs in progress, or has runs without a verdict.
731
+
732
+ Optimization trial runs test a changed configuration, so `eval=` and other
733
+ selectors leave them out. Add `trial=<trialId>` to compare one trial's runs,
734
+ including runs it reused. `gauge run list` shows the trial key and id.
735
+ `compare` prints the trial behind each side's trial runs. `gauge run list` and
736
+ `gauge run get` show a run's `optimizationTrial`.
737
+
738
+ To try other agents or sample counts once, without editing the saved measurement, pass
739
+ `--agent` and `--samples` to `gauge run --eval <id>` or `--preference <id>`.
740
+ The stored configuration stays unchanged.
741
+
742
+ `gauge run status <request-id>` prints a launch request's progress and
743
+ per-session results once, without waiting.
744
+
458
745
  ## Models, reports, and scripting
459
746
 
460
747
  Use `gauge models list` before pinning models. It reads the organization's live
@@ -462,10 +749,23 @@ catalog and shows every selectable logical model with its harness, provider,
462
749
  and effective capabilities. Open harnesses use `--agent pi:<model>` and
463
750
  `--agent opencode:<model>` just like the frontier harnesses.
464
751
 
465
- `gauge query` builds flexible reports over your event warehouse (a GA4
466
- `run_report` analog): pick a `--dataset`, group by `--dimensions`, and aggregate
467
- `--metrics`, or pass a whole spec with `--json`. `gauge query datasets` /
468
- `gauge query fields --dataset <name>` list what you can ask for.
752
+ `gauge query` uses SQLite locally and the reporting API when hosted. Both expose
753
+ `findings`; local also supports run summaries. Discover capabilities with
754
+ `gauge query datasets` and `gauge query fields --dataset findings` using the
755
+ selected backend. Unsupported datasets, filters, dimensions, and metrics fail
756
+ with actionable errors; unavailable data never becomes a fabricated zero.
757
+
758
+ ```sh
759
+ gauge query --local --dataset findings --dimensions kind,effect --metrics finding_count,affected_runs
760
+ gauge query --hosted --dataset findings --dimensions kind,effect --metrics finding_count,affected_runs
761
+ gauge query search "authentication failed" --eval <id> --local
762
+ ```
763
+
764
+ Findings queries use the current published analysis, so reanalysis is not double
765
+ counted. `finding_count` counts findings; `affected_runs` counts distinct runs.
766
+ Missing analysis differs from an analyzed run with no findings. Transcript search
767
+ scans selected evidence files and returns run/turn references. No transcript
768
+ index or background indexing process is maintained.
469
769
 
470
770
  Every read supports `-o json` for scripting. `GAUGE_API_TOKEN` overrides
471
771
  stored credentials (useful in CI). Exit codes: `0` ok · `1` error ·
@@ -509,13 +809,60 @@ preference:
509
809
  ```
510
810
 
511
811
  Write both brands in the prompt itself; metadata is only for analysis and never
512
- rewrites the task. `gauge evals verify`, `plan`, and `run` support mixed experience
513
- and preference suites. These fields require the CLI version containing this
514
- feature; CLI 0.16.0 and earlier reject them.
812
+ rewrites the task. `gauge run --verify`, `--dry-run`, and `gauge run` support mixed experience
813
+ and preference cases.
515
814
 
516
815
  Preference uses the existing app analyst: ranked picks for open-ended cases,
517
816
  and a winner (or neither) for head-to-head. Results are advisory. Preference-only
518
- GitHub checks finish neutral; experience criteria still gate mixed suites.
519
- `gauge run-requests wait` exits successfully for `MEASURED` or `PASSED`;
520
- input, execution, and analysis errors remain errors. Results describe the
817
+ GitHub checks finish neutral; experience criteria still gate mixed case sets.
818
+ `gauge run wait --check` exits successfully for `MEASURED` or `PASSED`;
819
+ input, execution, and analysis errors exit 1. Results describe the
521
820
  candidate revision and do not infer a base/head delta.
821
+
822
+ ## Local process lifecycle
823
+
824
+ `gauge run --local` runs in the foreground. Ctrl+C (or SIGTERM) stops the
825
+ queue, cancels active sessions, waits for cleanup, and exits with code 130.
826
+ Partial results and traces remain under `.gauge/runs`. Successful sessions
827
+ remove their containers and temporary credentials; the Docker image stays cached.
828
+
829
+ Use `gauge run --detach --local` to keep a batch running after the CLI exits.
830
+
831
+ Use `gauge run --local --runtime-image my-runtime:dev` to select a compatible
832
+ custom image, including with `--dry-run` or `--detach`. Derive from the standard
833
+ runtime to add Python or other system dependencies; see the
834
+ [Python example](https://github.com/gauge-sh/gauge-cli/tree/main/examples/runtime-python).
835
+ Preserve the runtime's tools, contract label, PATH, host-user permissions, and
836
+ entrypoint behavior. Gauge validates
837
+ the image before accepting a batch and pins its immutable ID for queued sessions.
838
+ A detached supervisor drains the queue and exits when the batch settles. The
839
+ launch prints its batch ID and these commands, to run from the same repository:
840
+
841
+ ```sh
842
+ gauge run status --batch <batch-id> --local
843
+ gauge run wait --batch <batch-id> --local
844
+ gauge run cancel --batch <batch-id> --local
845
+ ```
846
+
847
+ `gauge run wait --batch <batch-id> --local` blocks until the batch settles
848
+ (`completed`, `canceled`, `failed`, or `interrupted`), then prints each
849
+ session's state and the next step. It exits 0 once settled; `--check` exits 1
850
+ unless every session completed.
851
+
852
+ `gauge run status <run-id> --local` inspects a single session;
853
+ `gauge run cancel <run-id> --local` cancels only that run. All support `-o json`.
854
+ Batch cancellation is persisted before workers are contacted, including during
855
+ startup. A pending cancellation returns nonzero; inspect status before retrying.
856
+
857
+ After an interruption, `gauge cleanup --local` retries cleanup for recorded
858
+ interrupted sessions or sessions with incomplete cleanup in this repository.
859
+ It preserves evidence and recovered work, and makes recovered results visible
860
+ in `gauge run list`. It leaves live sessions, other repositories, unrelated
861
+ containers, and cached images alone. Recovery checks Gauge's container label
862
+ and workspace mount before removing a remaining container. Uncertain worker
863
+ ownership is reported rather than signaling a potentially reused PID.
864
+
865
+ Local workers have a 30-minute turn deadline and a 60-second supervisor
866
+ response deadline, so losing a supervisor cannot leave an idle worker forever.
867
+ Cleanup cannot discover sessions whose `.gauge` records were deleted; keep
868
+ those records until cleanup has succeeded.