@withgauge/cli 0.21.0 → 0.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +427 -80
- package/dist/batchWorker.js +2 -0
- package/dist/chunk-FCK3MQPV.js +2 -0
- package/dist/chunk-KLWQOTEK.js +3 -0
- package/dist/chunk-PSBF4TEK.js +1 -0
- package/dist/chunk-RB7Q72OV.js +1 -0
- package/dist/chunk-RI75U3EB.js +70 -0
- package/dist/index.js +564 -11697
- package/dist/telemetryWorker.js +1 -0
- package/dist/worker.js +1 -0
- package/package.json +13 -4
package/README.md
CHANGED
|
@@ -7,31 +7,220 @@ live from your terminal.
|
|
|
7
7
|
```sh
|
|
8
8
|
npm install -g @withgauge/cli
|
|
9
9
|
gauge update # update the CLI to the latest published version
|
|
10
|
-
gauge onboard # sign in
|
|
10
|
+
gauge onboard # sign in and create a workspace (account only)
|
|
11
11
|
gauge auth login # (or just sign in)
|
|
12
12
|
gauge orgs use <org> # set your default organization
|
|
13
|
+
gauge init # initialize a workspace; local is the default
|
|
14
|
+
gauge init --hosted # set a private hosted backend preference
|
|
15
|
+
gauge init --cases # optionally scaffold shared gauge-cases/
|
|
13
16
|
```
|
|
14
17
|
|
|
15
|
-
Requires Node.js ≥ 22.
|
|
18
|
+
Requires Node.js ≥ 22.13.
|
|
16
19
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
+
## Workspace and local storage
|
|
21
|
+
|
|
22
|
+
`gauge init --local` works without Git. It writes a minimal shared manifest:
|
|
23
|
+
|
|
24
|
+
```json
|
|
25
|
+
{ "version": 2, "inputs": {} }
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
The private `.gauge/config.json` contains `{ "version": 1, "backend": "local" }`.
|
|
29
|
+
`.gauge/.gitignore` ignores all workspace state. SQLite in `.gauge/gauge.sqlite`
|
|
30
|
+
stores saved measurements, revisions, launch/session summaries, and published
|
|
31
|
+
analyses. Raw transcripts, normalized events, diffs, and diagnostics remain under
|
|
32
|
+
`.gauge/runs/`; input blobs and batch lifecycle files remain alongside them.
|
|
33
|
+
|
|
34
|
+
Moving the workspace preserves identity. Fresh clones or linked worktrees start
|
|
35
|
+
independently; switching Git branches keeps local history. Do not treat `.gauge/`
|
|
36
|
+
as a disposable cache. Credentials and user preferences stay user-level.
|
|
37
|
+
`gauge run repair --local` rebuilds session summaries from durable artifacts while
|
|
38
|
+
preserving authored measurements and analyses. It does not rerun sessions or judges.
|
|
39
|
+
|
|
40
|
+
## Analyze local sessions
|
|
41
|
+
|
|
42
|
+
Your coding agent reads evidence and publishes a structured analysis:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
gauge instructions session-analysis --local
|
|
46
|
+
gauge run analysis schema --local > analysis.schema.json
|
|
47
|
+
gauge run get <run-id> --logs --local -o json
|
|
48
|
+
# Author analysis.json against the printed schema and session evidence.
|
|
49
|
+
gauge run analysis save <run-id> --file analysis.json --local
|
|
50
|
+
gauge run insight <run-id> --local
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Publication validates the versioned schema, criterion coverage, and evidence
|
|
54
|
+
references, then saves findings, product choices, and criterion judgments in one
|
|
55
|
+
transaction. Reanalysis creates a revision and updates the current analysis.
|
|
56
|
+
`run insight` leads with Friction, Objections, and Opportunities in either backend.
|
|
57
|
+
A Markdown file alone does not mark a session analyzed. Unknown future schema
|
|
58
|
+
versions produce an upgrade error without altering saved data. Database schema
|
|
59
|
+
migrations preserve analyses and never silently reinterpret old judgments.
|
|
60
|
+
|
|
61
|
+
## Local Docker runtime
|
|
62
|
+
|
|
63
|
+
Local runs need Docker running Linux containers and an agent credential. Linux
|
|
64
|
+
is the container OS, not a requirement for your host: Docker Desktop on macOS
|
|
65
|
+
provides a Linux daemon and shares the local files used by sessions.
|
|
66
|
+
Install the runtime without a Gauge account or a source checkout:
|
|
67
|
+
|
|
68
|
+
```sh
|
|
69
|
+
gauge local runtime pull
|
|
70
|
+
gauge local runtime inspect -o json
|
|
71
|
+
gauge run --local --dry-run
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Each CLI release pins its tested public `ghcr.io/gauge-sh/gauge-runtime` image by
|
|
75
|
+
immutable digest. Inspection reports the reference, local image ID, registry
|
|
76
|
+
digest, contract, platform, and readiness errors. `ready` covers local image
|
|
77
|
+
metadata; agent credential checks happen separately. Neither inspection nor a
|
|
78
|
+
dry run downloads an image. `pull` explicitly downloads the supported image;
|
|
79
|
+
previously pulled images can be reused offline (agent calls still need network
|
|
80
|
+
access). Pull progress goes to stderr so JSON output remains parseable.
|
|
81
|
+
|
|
82
|
+
The runtime includes native `linux/amd64` and `linux/arm64` images. Docker selects
|
|
83
|
+
the matching image on Intel/AMD or Apple Silicon/ARM machines. No architecture
|
|
84
|
+
emulation is needed. Docker must be able to bind-mount this machine's temporary
|
|
85
|
+
and input directories; an unrelated remote Docker server cannot access them.
|
|
86
|
+
On Windows, run the CLI inside WSL because Gauge uses Unix sockets and UID/GID.
|
|
87
|
+
|
|
88
|
+
## Local agent credentials
|
|
89
|
+
|
|
90
|
+
Local provider credentials are separate from a Gauge account and hosted credential
|
|
91
|
+
brokering. Use native harness logins or configure a named environment reference:
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
gauge local profiles set work --agent CODEX_CLI --provider openai --auth api_key --env TEAM_OPENAI_KEY
|
|
95
|
+
gauge local profiles use work
|
|
96
|
+
gauge local profiles inspect work -o json
|
|
97
|
+
gauge run --local --profile work --dry-run -o json
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
`--env` takes the variable's name, never its secret value. Profiles store only
|
|
101
|
+
references; list and inspect output do not expose credentials. Update a profile
|
|
102
|
+
with `set ... --replace` or remove it with `remove <profile>` without deleting
|
|
103
|
+
the native login. Local readiness checks do not verify provider acceptance or
|
|
104
|
+
model access. Use `gauge local profiles --help` for profile commands and
|
|
105
|
+
`gauge instructions --local` for the local workflow.
|
|
106
|
+
|
|
107
|
+
## Anonymous CLI telemetry
|
|
108
|
+
|
|
109
|
+
The installed CLI collects command usage by default to help improve Gauge.
|
|
110
|
+
Opt out for your user account or for a single process:
|
|
20
111
|
|
|
21
112
|
```sh
|
|
22
|
-
|
|
113
|
+
gauge telemetry disable
|
|
114
|
+
gauge telemetry status
|
|
115
|
+
gauge telemetry enable
|
|
116
|
+
GAUGE_TELEMETRY_DISABLED=1 gauge run --verify
|
|
23
117
|
```
|
|
24
118
|
|
|
119
|
+
The environment variable overrides the saved preference. These controls work
|
|
120
|
+
without an account or network access and support `-o json`. They apply to CLI
|
|
121
|
+
command telemetry; authenticated Gauge API requests have separate server-side
|
|
122
|
+
usage analytics.
|
|
123
|
+
|
|
124
|
+
Each `cli_command_invoked` event contains only the canonical command name
|
|
125
|
+
(for example, `run list`), its UTC invocation timestamp, CLI version, and a
|
|
126
|
+
random installation UUID. Arguments, flag values, file paths, prompts, file
|
|
127
|
+
contents, environment variables, errors, and account identifiers are excluded.
|
|
128
|
+
The UUID is independent of login and is never linked to your Gauge account.
|
|
129
|
+
This is pseudonymous usage tracking: an installation can be recognized across
|
|
130
|
+
commands, but no personal identity is sent. HTTP requests necessarily expose a
|
|
131
|
+
source IP to the receiving infrastructure; event properties mask IP addresses,
|
|
132
|
+
disable GeoIP enrichment, and disable person-profile creation.
|
|
133
|
+
|
|
134
|
+
Preferences live in `~/.config/gauge/telemetry.json` and the UUID in
|
|
135
|
+
`~/.config/gauge/telemetry-id` (respecting `XDG_CONFIG_HOME`). They are shared
|
|
136
|
+
across repositories, independently of each repository's `.gauge/` results.
|
|
137
|
+
Deleting `telemetry-id` resets the identity on the next collected command.
|
|
138
|
+
Consumers are installations, not necessarily individual people. Session
|
|
139
|
+
analysis can group an installation's events using a 30-minute inactivity gap;
|
|
140
|
+
the CLI does not generate a different identity for each invocation.
|
|
141
|
+
|
|
142
|
+
Collection occurs when a parsed command is about to run, including commands
|
|
143
|
+
whose handlers subsequently fail. Help, version, parsing errors, telemetry
|
|
144
|
+
controls, and source development runs are excluded. A detached worker handles
|
|
145
|
+
all telemetry storage and network access, with a two-second request timeout
|
|
146
|
+
and a three-second worker lifetime. Commands never wait for delivery, and
|
|
147
|
+
failures are silent. There are no retries or offline queue, so events may be
|
|
148
|
+
lost. Corrupt or unreadable preferences suppress collection until repaired.
|
|
149
|
+
|
|
150
|
+
Events go to Gauge's existing PostHog project using its public ingestion key.
|
|
151
|
+
The [capture API](https://posthog.com/docs/api/capture) and
|
|
152
|
+
[anonymous event controls](https://posthog.com/docs/product-analytics/capture-events#advanced-anonymous-vs-identified-events)
|
|
153
|
+
describe the transport and person-profile behavior.
|
|
154
|
+
|
|
155
|
+
## Local or hosted
|
|
156
|
+
|
|
157
|
+
For an end-to-end improvement loop, run `gauge instructions hillclimb`. It guides
|
|
158
|
+
your coding agent to identify product inputs, propose a representative pilot,
|
|
159
|
+
inspect and publish findings, implement a focused change, and rerun to compare.
|
|
160
|
+
Init and the generated `AGENTS.md` point to this guidance. `gauge install-skill`
|
|
161
|
+
installs the Gauge skill, which directs agents to the installed CLI's instructions.
|
|
162
|
+
|
|
163
|
+
Execution backend is independent of definition source. Resolve an explicit
|
|
164
|
+
`--local`/`--hosted` flag, then `.gauge/config.json`, user preference, and finally
|
|
165
|
+
local. Signing in does not change the backend. `gauge.json` is the shared project
|
|
166
|
+
manifest for named inputs, optional case discovery, and GitHub checks. It never
|
|
167
|
+
contains the execution preference. Git is optional for workspace initialization.
|
|
168
|
+
|
|
169
|
+
| Workflow | Local | Hosted |
|
|
170
|
+
|---|---|---|
|
|
171
|
+
| Skill, npm package (CLI or SDK), and `files` inputs | yes | yes |
|
|
172
|
+
| Docs supplied as `files` inputs | yes | yes |
|
|
173
|
+
| Eval and Agent Preference cases in `gauge-cases/` | yes (you read the sessions) | yes (judged) |
|
|
174
|
+
| `run`, `run list`, `run compare --local`, `models list`, `cleanup`, `feedback` | yes | yes |
|
|
175
|
+
| Binary, website, and MCP inputs | no | yes |
|
|
176
|
+
| Saved eval sets and preference prompts | yes | yes |
|
|
177
|
+
| Project inputs and previews on saved measurement launches | yes | use case files/specs |
|
|
178
|
+
| Findings and summary queries | yes, supported fields | yes |
|
|
179
|
+
| Personas, recurring schedules, PR checks | no | yes |
|
|
180
|
+
| Optimizations (managed docs rewriting) | no | yes |
|
|
181
|
+
| Stored credentials for MCP servers and connections | no | yes |
|
|
182
|
+
| Additional reporting datasets, dashboards, org settings | no | yes |
|
|
183
|
+
|
|
184
|
+
Local runs need Docker and the agent's own login (`claude` or
|
|
185
|
+
`ANTHROPIC_API_KEY` for Claude Code; `codex login` for Codex, Pi, and
|
|
186
|
+
OpenCode). They do not need a Gauge account. Hosted runs need a Gauge account
|
|
187
|
+
(`gauge onboard`) and use credits. In local mode, a hosted-only command says so
|
|
188
|
+
and prints the next step.
|
|
189
|
+
|
|
190
|
+
For coding agents, we recommend the
|
|
191
|
+
[`gauge-agents` skill](https://github.com/gauge-sh/gauge-skills/tree/main/gauge-agents).
|
|
192
|
+
It makes Gauge CLI guidance and repository setup workflows discoverable in future
|
|
193
|
+
sessions. Install it separately from repo setup, running from your project root:
|
|
194
|
+
|
|
195
|
+
```sh
|
|
196
|
+
gauge install-skill
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
In a terminal, select your coding agent and installation scope in the installer.
|
|
200
|
+
Project scope adds skill files to this repo; global scope makes the skill available
|
|
201
|
+
across projects. The command runs the pinned `skills@1.5.0` installer through npx
|
|
202
|
+
and forwards its prompts and output. When a coding agent runs it, the command
|
|
203
|
+
installs for that agent at project scope without prompts and prints a short summary.
|
|
204
|
+
To choose the agent or scope explicitly:
|
|
205
|
+
|
|
206
|
+
```sh
|
|
207
|
+
gauge install-skill --agent claude-code --yes # this project
|
|
208
|
+
gauge install-skill --agent codex --global --yes # across projects
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
Repeat `--agent` to target multiple agents. `--yes` requires an explicit agent
|
|
212
|
+
target; `--copy` copies files instead of creating symlinks.
|
|
213
|
+
|
|
25
214
|
## Usage
|
|
26
215
|
|
|
27
216
|
```sh
|
|
28
217
|
gauge status # the org at a glance
|
|
29
|
-
gauge
|
|
30
|
-
gauge preference list # Agent Preference prompts
|
|
218
|
+
gauge eval list # saved eval sets
|
|
219
|
+
gauge preference list # saved Agent Preference prompts
|
|
31
220
|
gauge models list # supported model × harness × provider targets
|
|
32
|
-
gauge
|
|
33
|
-
gauge
|
|
34
|
-
gauge
|
|
221
|
+
gauge eval edit <id> --cadence weekly --samples 3 # when and how much it runs
|
|
222
|
+
gauge run --eval <id> # launch now (ORG-funded, credit-gated)
|
|
223
|
+
gauge run watch <run-id> # follow a run as the agent works
|
|
35
224
|
gauge stats rankings -o json # brand rankings (share of voice) as JSON
|
|
36
225
|
gauge query -d ecosystem -m installs,install_rate -o json # a flexible report
|
|
37
226
|
```
|
|
@@ -39,27 +228,28 @@ gauge query -d ecosystem -m installs,install_rate -o json # a flexible report
|
|
|
39
228
|
`gauge instructions` prints a Gauge overview, CLI setup and usage rules, and
|
|
40
229
|
roughly 100 words of essential guidance per module. Each summary points to
|
|
41
230
|
`gauge instructions <topic>` for the full page. `gauge --help` points agents at
|
|
42
|
-
it
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
Use `gauge instructions
|
|
47
|
-
model selection, observable judging criteria, and a complete
|
|
48
|
-
Use `gauge instructions preference` to
|
|
49
|
-
|
|
231
|
+
it. Install the Gauge skill with `gauge install-skill` to make this guidance
|
|
232
|
+
discoverable in future sessions. Read instructions from the installed CLI so
|
|
233
|
+
they match its commands.
|
|
234
|
+
|
|
235
|
+
Use `gauge instructions run` before writing evals or starting runs: it covers
|
|
236
|
+
developer tasks, model selection, observable judging criteria, and a complete
|
|
237
|
+
migration example. Use `gauge instructions eval` and `gauge instructions preference` to save eval sets and
|
|
238
|
+
preference prompts, and `gauge instructions session-analysis` to investigate
|
|
239
|
+
product choices, source exposure, and interventions, from individual traces to
|
|
240
|
+
larger cohorts.
|
|
50
241
|
Use `gauge instructions optimization` for the full optimization workflow.
|
|
51
242
|
`gauge instructions --list` shows available topics.
|
|
52
|
-
`gauge instructions evals --install` installs the eval guidance as a local skill.
|
|
53
243
|
|
|
54
244
|
Run `gauge --help` (or `gauge <command> --help`) for the full command set:
|
|
55
|
-
|
|
56
|
-
|
|
245
|
+
run, eval, preference, models, personas, skills, mcp, repos, optimizations,
|
|
246
|
+
brands, tags, actions, dashboards, members, org, provider keys,
|
|
57
247
|
billing, usage, stats, query, and more.
|
|
58
248
|
|
|
59
249
|
## Run committed Markdown evals
|
|
60
250
|
|
|
61
251
|
File evals keep the task, run settings, and judged criteria in Git. Create
|
|
62
|
-
`gauge-
|
|
252
|
+
`gauge-cases/healthcheck.md` with YAML frontmatter and a Markdown task body:
|
|
63
253
|
|
|
64
254
|
```markdown
|
|
65
255
|
---
|
|
@@ -76,19 +266,28 @@ config:
|
|
|
76
266
|
Add a /health endpoint and run the relevant app checks.
|
|
77
267
|
```
|
|
78
268
|
|
|
79
|
-
Check your setup, then
|
|
269
|
+
Check your setup, then preview and run the current working-tree cases and inputs:
|
|
80
270
|
|
|
81
271
|
```sh
|
|
82
|
-
gauge
|
|
83
|
-
gauge
|
|
84
|
-
gauge
|
|
85
|
-
gauge
|
|
86
|
-
gauge run
|
|
272
|
+
gauge run --verify # local; includes uncommitted working-tree edits
|
|
273
|
+
gauge run --verify -o json # versioned global/per-input yes/no/unknown checklist
|
|
274
|
+
gauge run --dry-run
|
|
275
|
+
gauge run --yes
|
|
276
|
+
gauge run wait <request-id> -o json
|
|
87
277
|
```
|
|
88
278
|
|
|
89
|
-
Cases default to `gauge-
|
|
90
|
-
that convention for `
|
|
91
|
-
|
|
279
|
+
Cases default to `gauge-cases/**/*.md`. `--files` accepts quoted paths/globs and overrides
|
|
280
|
+
that convention for `--dry-run` and `run`. Interactive CLI launches snapshot current
|
|
281
|
+
working-tree definitions in either backend, including uncommitted edits. Selected
|
|
282
|
+
filesystem inputs are packaged for local execution or hosted upload; explicit remote
|
|
283
|
+
artifacts retain their declared source. GitHub checks fetch committed definitions.
|
|
284
|
+
Content digests identify the exact bytes separately from Git provenance.
|
|
285
|
+
Preview `inputSnapshots` includes the selected source `path`,
|
|
286
|
+
`gaugeArchive: { sha256, bytes }`, and the archived `files` (the first 100) with
|
|
287
|
+
`fileCount`. Those values describe Gauge's delivery archive, which contains the
|
|
288
|
+
selected files. `unattachedInputs` names the `gauge.json` inputs that the launch
|
|
289
|
+
leaves out. For npm inputs it contains the `.tgz`; the raw
|
|
290
|
+
tarball has a different hash and size. Each launch takes a fresh snapshot.
|
|
92
291
|
Each case owns its agents, models, consumer repository/ref, Persona and criteria.
|
|
93
292
|
The optional `config.inputs` selects named inputs; all attach by default.
|
|
94
293
|
|
|
@@ -107,7 +306,7 @@ account settings are outside this command's scope and do not create unknown rows
|
|
|
107
306
|
Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
|
|
108
307
|
No workflow code is executed.
|
|
109
308
|
|
|
110
|
-
`gauge
|
|
309
|
+
`gauge run --verify -o json` writes exactly one JSON object to stdout, including on
|
|
111
310
|
validation failure. The versioned report provides:
|
|
112
311
|
|
|
113
312
|
- `version: 1` and `mode: "local-static"` identify the report contract and scope.
|
|
@@ -125,7 +324,7 @@ validation failure. The versioned report provides:
|
|
|
125
324
|
For example, an agent can select unresolved checks with:
|
|
126
325
|
|
|
127
326
|
```sh
|
|
128
|
-
gauge
|
|
327
|
+
gauge run --verify -o json | jq '{status, facts, global: [.global[] | select(.state != "yes")], inputs: [.inputs[] | {name, checks: [.checks[] | select(.state != "yes")]}]}'
|
|
129
328
|
```
|
|
130
329
|
|
|
131
330
|
An optional `gauge.json` declares inputs produced by your normal CI or committed
|
|
@@ -271,16 +470,24 @@ rerun Gauge to test the latest build.
|
|
|
271
470
|
With CLI 0.18.0 or later, supply built inputs directly to run without the GitHub App:
|
|
272
471
|
|
|
273
472
|
```sh
|
|
274
|
-
gauge --org my-org
|
|
275
|
-
gauge --org my-org run
|
|
473
|
+
gauge --org my-org run --input product-analytics=./dist/skill.zip --yes -o json > request.json
|
|
474
|
+
gauge --org my-org run wait "$(jq -er '.runRequestId' request.json)" --check -o json > results.json
|
|
276
475
|
```
|
|
277
476
|
|
|
477
|
+
`--check` fails the job unless the verdict is `PASSED` or `MEASURED`. Without
|
|
478
|
+
it, `run wait` exits 0 once the request settles, whatever the verdict.
|
|
479
|
+
|
|
278
480
|
Set `GAUGE_API_TOKEN` as a CI secret and replace `my-org` with your workspace slug.
|
|
279
481
|
This can run immediately after your build in the same CI job.
|
|
280
482
|
|
|
281
|
-
Repeat `--input name=path` for each input.
|
|
282
|
-
|
|
283
|
-
|
|
483
|
+
Repeat `--input name=path` for each input. The path replaces the input's
|
|
484
|
+
configured files entirely; only its type and settings carry over. It may be a
|
|
485
|
+
glob, using the same syntax as `gauge.json`; quote it so the shell passes it
|
|
486
|
+
through. Skills accept a directory containing `SKILL.md` or a ZIP/tar.gz archive;
|
|
487
|
+
npm inputs accept a packed `.tgz` or a glob matching exactly one. Files inputs
|
|
488
|
+
accept a directory, a single file, or a glob; archive paths are relative to the
|
|
489
|
+
directory, or to the glob's directory before its first wildcard. Binary inputs
|
|
490
|
+
accept a directory or glob, with `commands` relative to it.
|
|
284
491
|
Use a dedicated build directory: links are rejected, and common credential files
|
|
285
492
|
and `node_modules` are excluded. Each input is limited to 512 MiB and 20,000 files.
|
|
286
493
|
|
|
@@ -292,11 +499,32 @@ Website previews and private consumer repositories still require their configure
|
|
|
292
499
|
access. Public consumer repositories and the default workspace need no GitHub App.
|
|
293
500
|
|
|
294
501
|
`wait -o json` returns the verdict, session results, criterion explanations, and
|
|
295
|
-
session links. It exits
|
|
296
|
-
|
|
502
|
+
session links. It exits 0 once the request settles, whatever the verdict. Add
|
|
503
|
+
`--check` to exit 1 unless the verdict is `PASSED` or `MEASURED`; the results
|
|
504
|
+
JSON is still the only output. Automatic PR checks continue to use configured
|
|
297
505
|
sources; disable those checks in Gauge's GitHub settings if CI owns the trigger.
|
|
298
506
|
Save `results.json` as a CI artifact even when the check fails.
|
|
299
507
|
|
|
508
|
+
### Antigravity (hosted)
|
|
509
|
+
|
|
510
|
+
Antigravity is available to organizations enrolled in Antigravity access. Run
|
|
511
|
+
`gauge models list --hosted` to find the available model identifiers, then select
|
|
512
|
+
one with `--agent antigravity:<model>` when creating or launching a saved eval or
|
|
513
|
+
preference prompt. Case files and JSON specs use `agent: ANTIGRAVITY` with the
|
|
514
|
+
model identifier in `models`.
|
|
515
|
+
|
|
516
|
+
Launch with `--hosted`, for example:
|
|
517
|
+
|
|
518
|
+
```sh
|
|
519
|
+
gauge run --eval <id> --agent antigravity:<model> --hosted
|
|
520
|
+
```
|
|
521
|
+
|
|
522
|
+
For case files or a JSON case spec, preview with `gauge run --hosted --dry-run`
|
|
523
|
+
or `gauge run --apply cases.json --hosted --dry-run` before launching.
|
|
524
|
+
Hosted previews contact Gauge and may upload inputs; launches use credits.
|
|
525
|
+
The server checks organization access and model availability. Antigravity cannot
|
|
526
|
+
run locally yet.
|
|
527
|
+
|
|
300
528
|
### Authenticated MCP servers
|
|
301
529
|
|
|
302
530
|
Connect your account in Gauge's MCP setup, completing OAuth in your browser when
|
|
@@ -312,31 +540,43 @@ marker, and a readiness `pattern` with named `url` and `sha` captures. Gauge sel
|
|
|
312
540
|
candidate-specific evidence; timestamps alone cannot establish the revision.
|
|
313
541
|
Comments must belong to a unique open same-repository PR at the candidate head.
|
|
314
542
|
Previews must be public HTTPS origins, without authentication or path prefixes.
|
|
315
|
-
They are fetched live rather than snapshotted.
|
|
316
|
-
|
|
543
|
+
They are fetched live rather than snapshotted.
|
|
544
|
+
|
|
545
|
+
Canonical-site preview routing requires hosted execution. Local runs reject
|
|
546
|
+
website inputs, including `--input` overrides, before checking Docker or credentials.
|
|
547
|
+
For local content-only testing, supply docs as a `files` input or put a reachable
|
|
548
|
+
preview URL directly in the prompt. Inputs never include `.gauge/`. Confirm that the preview contains the edits
|
|
549
|
+
and is reachable from the agent's environment; the host's localhost is not
|
|
550
|
+
automatically reachable from a container. These alternatives change exposure and
|
|
551
|
+
do not test discovery of the canonical docs. Static verification validates
|
|
552
|
+
configuration; it does not prove backend support, delivery, or an eval pass.
|
|
317
553
|
|
|
318
554
|
Editor schemas are published at
|
|
319
|
-
`https://agents.withgauge.com/schemas/gauge/v2.
|
|
320
|
-
`https://agents.withgauge.com/schemas/gauge/v2.
|
|
555
|
+
`https://agents.withgauge.com/schemas/gauge/v2.8.0/gauge.json` and
|
|
556
|
+
`https://agents.withgauge.com/schemas/gauge/v2.8.0/case.json`. Associate them in editor
|
|
321
557
|
settings; `gauge.json` does not accept a `$schema` field. These release URLs are
|
|
322
|
-
immutable.
|
|
323
|
-
replacing `use` with input `type` and `path`/`paths`, and moving cases into
|
|
324
|
-
`gauge-evals/` (or setting `cases` explicitly).
|
|
558
|
+
immutable. Cases default to `gauge-cases/`, or use explicit `cases` paths.
|
|
325
559
|
|
|
326
560
|
`--org` overrides the committed organization. Root `cases` can change discovery;
|
|
327
|
-
root `version` defaults to 2. Saved evals use `gauge
|
|
328
|
-
use platform funding by default and can add `--bill-to-org`.
|
|
561
|
+
root `version` defaults to 2. Saved evals use `gauge run --eval <id>`.
|
|
329
562
|
|
|
330
563
|
## Measurements own their configuration
|
|
331
564
|
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
565
|
+
Saved eval sets and preference prompts provide a stable grouping for sessions
|
|
566
|
+
plus editable templates and defaults. Local records live in workspace SQLite;
|
|
567
|
+
hosted records live in your organization. Launches freeze identity, revision, and
|
|
568
|
+
effective settings. Renaming or editing a measurement preserves its history.
|
|
569
|
+
Local measurements run on demand; recurring schedules, personas, markets, and
|
|
570
|
+
hosted resource references require hosted execution. Measurements do not store
|
|
571
|
+
project inputs: attach them to a local launch with
|
|
572
|
+
`gauge run --eval <id> --input <name>`, or `--input <name>=<path>` to replace the
|
|
573
|
+
configured files.
|
|
335
574
|
|
|
336
|
-
An **eval** is one prompt against one configuration
|
|
575
|
+
An **eval** is one prompt against one configuration. This hosted example uses
|
|
576
|
+
registered skills and recurring scheduling; omit those hosted features locally:
|
|
337
577
|
|
|
338
578
|
```sh
|
|
339
|
-
gauge
|
|
579
|
+
gauge eval create --name "add a healthcheck" \
|
|
340
580
|
--text "Add a /health endpoint that returns 200 OK." \
|
|
341
581
|
--criterion "Returns 200: the endpoint responds 200 with no auth" \
|
|
342
582
|
--repo https://github.com/acme/api --ref main \
|
|
@@ -345,12 +585,14 @@ gauge evals create --name "add a healthcheck" \
|
|
|
345
585
|
--cadence weekly --samples 3
|
|
346
586
|
```
|
|
347
587
|
|
|
348
|
-
`gauge
|
|
588
|
+
`gauge eval edit <id>` changes any of it later; omitted flags keep their
|
|
349
589
|
stored value, and each `--clear-*` flag empties one field. To measure the same
|
|
350
590
|
prompt against a second repository or roster, copy the eval in the app and edit
|
|
351
591
|
the copy — one eval is always one configuration.
|
|
352
592
|
|
|
353
|
-
A **preference prompt** is a reusable question.
|
|
593
|
+
A **preference prompt** is a reusable question. The following hosted example
|
|
594
|
+
uses scenarios and recurring scheduling. Locally, save a prompt and launch it
|
|
595
|
+
on demand with `gauge run --preference <id> --local`. It owns one set of run
|
|
354
596
|
settings — roster, resources, cadence, samples — and any number of
|
|
355
597
|
**scenarios**, each one repository × persona cell. Every scenario runs with
|
|
356
598
|
the prompt's settings, so one prompt measures the same question across several
|
|
@@ -374,32 +616,30 @@ A prompt with zero scenarios is a valid definition — it simply never comes due
|
|
|
374
616
|
Sessions per launch = scenarios × agent/model targets × samples. Omit the
|
|
375
617
|
run-settings flags on `create` and the org's default roster applies.
|
|
376
618
|
|
|
377
|
-
##
|
|
619
|
+
## Hosted schedules
|
|
378
620
|
|
|
379
621
|
`--cadence none` is "manual": the measurement stops coming due and runs only
|
|
380
622
|
when you launch it. There is no separate pause or enabled flag, in the CLI or
|
|
381
623
|
the app. `--cadence daily|weekly|monthly` puts it back on a rhythm, and
|
|
382
|
-
`preference get
|
|
624
|
+
`gauge preference get <id>` reads a manual prompt back as `cadence: manual`.
|
|
383
625
|
|
|
384
|
-
`gauge
|
|
626
|
+
`gauge run --eval <id>` and `gauge run --preference <id>` are the separate
|
|
385
627
|
immediate launches. They use the stored configuration and ask before spending
|
|
386
628
|
(`--yes` skips the prompt for scripts).
|
|
387
|
-
`gauge preference
|
|
388
|
-
omit it and every scenario runs.
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
credits instead. Other users always use org credits and can launch only in orgs
|
|
392
|
-
they belong to.
|
|
629
|
+
`gauge run --preference <id> --scenario <scenarioId>` (repeatable) runs a subset;
|
|
630
|
+
omit it and every scenario runs. On a recurring cadence, an eval launch or a
|
|
631
|
+
full preference launch moves the next scheduled run to one period from now. A
|
|
632
|
+
scenario subset leaves the schedule unchanged.
|
|
393
633
|
|
|
394
|
-
## Optimizations
|
|
634
|
+
## Optimizations (hosted)
|
|
395
635
|
|
|
396
636
|
Start by finding runs that encountered the surface you want to improve:
|
|
397
637
|
|
|
398
638
|
```sh
|
|
399
639
|
gauge instructions optimization
|
|
400
640
|
gauge optimizations baselines --eval ev_1 -o json
|
|
401
|
-
gauge
|
|
402
|
-
gauge
|
|
641
|
+
gauge run insight run_1 -o json
|
|
642
|
+
gauge run get run_1 --logs -o json
|
|
403
643
|
gauge optimizations start --eval ev_1 --from-run run_1 --yes
|
|
404
644
|
gauge optimizations watch cl_1
|
|
405
645
|
```
|
|
@@ -434,7 +674,7 @@ Docs `REWRITE_DOCS` proposals supply `after.pages` with URL and exact `edits`
|
|
|
434
674
|
`source`: `scrape` means current fetched content, not the captured response.
|
|
435
675
|
|
|
436
676
|
For CLI trials, inspect `cliIntervention.guidance` and `targets` in authoring
|
|
437
|
-
and use `
|
|
677
|
+
and use `gauge run get <id> --logs` for the cited command evidence. `cli-change --file change.json`
|
|
438
678
|
accepts v3 `{schemaVersion:3, target, rules, shippingIntent}` and returns a sealed
|
|
439
679
|
bundle to put in `CLI_INTERVENTION.after`. It does not save or launch. The old
|
|
440
680
|
catalog/invocation-ID workflow is retired. Skills use `skill-files` and the
|
|
@@ -455,6 +695,53 @@ spending and `trials add --idempotency-key` when retrying single-trial writes.
|
|
|
455
695
|
Adoption selects a winner without changing the baseline or deploying the change;
|
|
456
696
|
`handoff` supplies instructions to implement it in your source repository.
|
|
457
697
|
|
|
698
|
+
## Compare runs
|
|
699
|
+
|
|
700
|
+
`gauge run compare` answers before/after questions in one call. Each side takes
|
|
701
|
+
repeatable `key=value` selectors: `request=`, `batch=`, `eval=`, `revision=`,
|
|
702
|
+
`experiment=`, `trial=`, `agent=`, `model=`, `since=`, and `until=`.
|
|
703
|
+
|
|
704
|
+
```sh
|
|
705
|
+
gauge run compare --baseline request=<id> --candidate request=<id>
|
|
706
|
+
gauge run compare --baseline eval=<id> --baseline until=2026-10-01 \
|
|
707
|
+
--candidate eval=<id> --candidate since=2026-10-01
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
It prints the pass rate, each criterion's pass rate, and median turns, cost,
|
|
711
|
+
and duration, for all runs and per agent and model. `-o json` returns the same
|
|
712
|
+
data with deltas.
|
|
713
|
+
|
|
714
|
+
With `--local` it compares local session summaries. Use saved measurement IDs
|
|
715
|
+
with `eval=` and `revision=<number>`, or select a `batch`, `agent`, `model`, and
|
|
716
|
+
time window. Compare preference runs by batch. The comparison uses current published
|
|
717
|
+
criterion judgments and recorded turns/duration; unknown metrics and unassessable
|
|
718
|
+
criteria stay out of their denominators. Completion is distinct from a judged result.
|
|
719
|
+
Inspect findings with `gauge run insight` and query capabilities with `gauge query fields`.
|
|
720
|
+
|
|
721
|
+
```sh
|
|
722
|
+
gauge run compare --local --baseline batch=<id> --candidate batch=<id>
|
|
723
|
+
```
|
|
724
|
+
|
|
725
|
+
Each run has an `evalRevision`: a hash of its prompt, repository, persona,
|
|
726
|
+
agent context, and criteria. `gauge eval edit` changes the revision of new
|
|
727
|
+
runs. Each case has its own revision. `gauge run list` shows it, and `compare`
|
|
728
|
+
warns when one eval or case ran under more than one revision; add
|
|
729
|
+
`revision=<hash>` to pick one. `compare` also warns when a side matched no
|
|
730
|
+
runs, has runs in progress, or has runs without a verdict.
|
|
731
|
+
|
|
732
|
+
Optimization trial runs test a changed configuration, so `eval=` and other
|
|
733
|
+
selectors leave them out. Add `trial=<trialId>` to compare one trial's runs,
|
|
734
|
+
including runs it reused. `gauge run list` shows the trial key and id.
|
|
735
|
+
`compare` prints the trial behind each side's trial runs. `gauge run list` and
|
|
736
|
+
`gauge run get` show a run's `optimizationTrial`.
|
|
737
|
+
|
|
738
|
+
To try other agents or sample counts once, without editing the saved measurement, pass
|
|
739
|
+
`--agent` and `--samples` to `gauge run --eval <id>` or `--preference <id>`.
|
|
740
|
+
The stored configuration stays unchanged.
|
|
741
|
+
|
|
742
|
+
`gauge run status <request-id>` prints a launch request's progress and
|
|
743
|
+
per-session results once, without waiting.
|
|
744
|
+
|
|
458
745
|
## Models, reports, and scripting
|
|
459
746
|
|
|
460
747
|
Use `gauge models list` before pinning models. It reads the organization's live
|
|
@@ -462,10 +749,23 @@ catalog and shows every selectable logical model with its harness, provider,
|
|
|
462
749
|
and effective capabilities. Open harnesses use `--agent pi:<model>` and
|
|
463
750
|
`--agent opencode:<model>` just like the frontier harnesses.
|
|
464
751
|
|
|
465
|
-
`gauge query`
|
|
466
|
-
`
|
|
467
|
-
|
|
468
|
-
|
|
752
|
+
`gauge query` uses SQLite locally and the reporting API when hosted. Both expose
|
|
753
|
+
`findings`; local also supports run summaries. Discover capabilities with
|
|
754
|
+
`gauge query datasets` and `gauge query fields --dataset findings` using the
|
|
755
|
+
selected backend. Unsupported datasets, filters, dimensions, and metrics fail
|
|
756
|
+
with actionable errors; unavailable data never becomes a fabricated zero.
|
|
757
|
+
|
|
758
|
+
```sh
|
|
759
|
+
gauge query --local --dataset findings --dimensions kind,effect --metrics finding_count,affected_runs
|
|
760
|
+
gauge query --hosted --dataset findings --dimensions kind,effect --metrics finding_count,affected_runs
|
|
761
|
+
gauge query search "authentication failed" --eval <id> --local
|
|
762
|
+
```
|
|
763
|
+
|
|
764
|
+
Findings queries use the current published analysis, so reanalysis is not double
|
|
765
|
+
counted. `finding_count` counts findings; `affected_runs` counts distinct runs.
|
|
766
|
+
Missing analysis differs from an analyzed run with no findings. Transcript search
|
|
767
|
+
scans selected evidence files and returns run/turn references. No transcript
|
|
768
|
+
index or background indexing process is maintained.
|
|
469
769
|
|
|
470
770
|
Every read supports `-o json` for scripting. `GAUGE_API_TOKEN` overrides
|
|
471
771
|
stored credentials (useful in CI). Exit codes: `0` ok · `1` error ·
|
|
@@ -509,13 +809,60 @@ preference:
|
|
|
509
809
|
```
|
|
510
810
|
|
|
511
811
|
Write both brands in the prompt itself; metadata is only for analysis and never
|
|
512
|
-
rewrites the task. `gauge
|
|
513
|
-
and preference
|
|
514
|
-
feature; CLI 0.16.0 and earlier reject them.
|
|
812
|
+
rewrites the task. `gauge run --verify`, `--dry-run`, and `gauge run` support mixed experience
|
|
813
|
+
and preference cases.
|
|
515
814
|
|
|
516
815
|
Preference uses the existing app analyst: ranked picks for open-ended cases,
|
|
517
816
|
and a winner (or neither) for head-to-head. Results are advisory. Preference-only
|
|
518
|
-
GitHub checks finish neutral; experience criteria still gate mixed
|
|
519
|
-
`gauge run
|
|
520
|
-
input, execution, and analysis errors
|
|
817
|
+
GitHub checks finish neutral; experience criteria still gate mixed case sets.
|
|
818
|
+
`gauge run wait --check` exits successfully for `MEASURED` or `PASSED`;
|
|
819
|
+
input, execution, and analysis errors exit 1. Results describe the
|
|
521
820
|
candidate revision and do not infer a base/head delta.
|
|
821
|
+
|
|
822
|
+
## Local process lifecycle
|
|
823
|
+
|
|
824
|
+
`gauge run --local` runs in the foreground. Ctrl+C (or SIGTERM) stops the
|
|
825
|
+
queue, cancels active sessions, waits for cleanup, and exits with code 130.
|
|
826
|
+
Partial results and traces remain under `.gauge/runs`. Successful sessions
|
|
827
|
+
remove their containers and temporary credentials; the Docker image stays cached.
|
|
828
|
+
|
|
829
|
+
Use `gauge run --detach --local` to keep a batch running after the CLI exits.
|
|
830
|
+
|
|
831
|
+
Use `gauge run --local --runtime-image my-runtime:dev` to select a compatible
|
|
832
|
+
custom image, including with `--dry-run` or `--detach`. Derive from the standard
|
|
833
|
+
runtime to add Python or other system dependencies; see the
|
|
834
|
+
[Python example](https://github.com/gauge-sh/gauge-cli/tree/main/examples/runtime-python).
|
|
835
|
+
Preserve the runtime's tools, contract label, PATH, host-user permissions, and
|
|
836
|
+
entrypoint behavior. Gauge validates
|
|
837
|
+
the image before accepting a batch and pins its immutable ID for queued sessions.
|
|
838
|
+
A detached supervisor drains the queue and exits when the batch settles. The
|
|
839
|
+
launch prints its batch ID and these commands, to run from the same repository:
|
|
840
|
+
|
|
841
|
+
```sh
|
|
842
|
+
gauge run status --batch <batch-id> --local
|
|
843
|
+
gauge run wait --batch <batch-id> --local
|
|
844
|
+
gauge run cancel --batch <batch-id> --local
|
|
845
|
+
```
|
|
846
|
+
|
|
847
|
+
`gauge run wait --batch <batch-id> --local` blocks until the batch settles
|
|
848
|
+
(`completed`, `canceled`, `failed`, or `interrupted`), then prints each
|
|
849
|
+
session's state and the next step. It exits 0 once settled; `--check` exits 1
|
|
850
|
+
unless every session completed.
|
|
851
|
+
|
|
852
|
+
`gauge run status <run-id> --local` inspects a single session;
|
|
853
|
+
`gauge run cancel <run-id> --local` cancels only that run. All support `-o json`.
|
|
854
|
+
Batch cancellation is persisted before workers are contacted, including during
|
|
855
|
+
startup. A pending cancellation returns nonzero; inspect status before retrying.
|
|
856
|
+
|
|
857
|
+
After an interruption, `gauge cleanup --local` retries cleanup for recorded
|
|
858
|
+
interrupted sessions or sessions with incomplete cleanup in this repository.
|
|
859
|
+
It preserves evidence and recovered work, and makes recovered results visible
|
|
860
|
+
in `gauge run list`. It leaves live sessions, other repositories, unrelated
|
|
861
|
+
containers, and cached images alone. Recovery checks Gauge's container label
|
|
862
|
+
and workspace mount before removing a remaining container. Uncertain worker
|
|
863
|
+
ownership is reported rather than signaling a potentially reused PID.
|
|
864
|
+
|
|
865
|
+
Local workers have a 30-minute turn deadline and a 60-second supervisor
|
|
866
|
+
response deadline, so losing a supervisor cannot leave an idle worker forever.
|
|
867
|
+
Cleanup cannot discover sessions whose `.gauge` records were deleted; keep
|
|
868
|
+
those records until cleanup has succeeded.
|