@withgauge/cli 0.17.1 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +114 -72
- package/dist/index.js +2092 -1597
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -76,7 +76,7 @@ config:
|
|
|
76
76
|
Add a /health endpoint and run the relevant app checks.
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
-
Check your
|
|
79
|
+
Check your setup, then commit the files before planning and running with Git inputs:
|
|
80
80
|
|
|
81
81
|
```sh
|
|
82
82
|
gauge evals verify # offline; includes uncommitted working-tree edits
|
|
@@ -87,7 +87,8 @@ gauge run-requests wait <request-id> -o json
|
|
|
87
87
|
```
|
|
88
88
|
|
|
89
89
|
Cases default to `gauge-evals/**/*.md`. `--files` accepts quoted paths/globs and overrides
|
|
90
|
-
that convention for `plan` and `run
|
|
90
|
+
that convention for `plan` and `run`. By default, these commands read committed Git
|
|
91
|
+
HEAD. With `--input`, they read your current files, including uncommitted edits.
|
|
91
92
|
Each case owns its agents, models, consumer repository/ref, Persona and criteria.
|
|
92
93
|
The optional `config.inputs` selects named inputs; all attach by default.
|
|
93
94
|
|
|
@@ -104,7 +105,7 @@ event filters and runtime conditions are not evaluated against a candidate PR.
|
|
|
104
105
|
Live input delivery, artifact contents, sandbox execution, App activation, and
|
|
105
106
|
account settings are outside this command's scope and do not create unknown rows.
|
|
106
107
|
Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
|
|
107
|
-
No workflow code is executed.
|
|
108
|
+
No workflow code is executed.
|
|
108
109
|
|
|
109
110
|
`gauge evals verify -o json` writes exactly one JSON object to stdout, including on
|
|
110
111
|
validation failure. The versioned report provides:
|
|
@@ -150,15 +151,19 @@ alongside your product:
|
|
|
150
151
|
}
|
|
151
152
|
```
|
|
152
153
|
|
|
153
|
-
Gauge waits for
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
154
|
+
Gauge waits for your PR build and tests its uploaded artifacts. Keep the workflow's
|
|
155
|
+
existing checkout settings, including GitHub's default merge checkout. Omit
|
|
156
|
+
`source.artifact` when the workflow uploads one artifact, or select it by name.
|
|
157
|
+
|
|
158
|
+
For automatic checks, connect the Gauge GitHub App with repository access and
|
|
159
|
+
Actions read permission. Set `org` and enable the repository and target branches
|
|
160
|
+
in **Settings → GitHub**. Waiting for inputs starts no sessions and spends no
|
|
161
|
+
session credits.
|
|
162
|
+
Fork pull requests are disabled by default. Check **Allow fork pull requests** to let
|
|
163
|
+
workspace members authorize an evaluation from its GitHub check. Authorization
|
|
164
|
+
covers one PR revision and uses workspace credits; new revisions and explicit
|
|
165
|
+
reruns require fresh approval. Configured inputs determine when an authorized
|
|
166
|
+
evaluation is ready to run.
|
|
162
167
|
A failed run or absent/expired artifact returns an actionable input failure.
|
|
163
168
|
Absent runs have a five-minute discovery grace period; known running or queued
|
|
164
169
|
CI can continue past an hour. Abandoned requests expire after seven days. Reruns retain captured bytes. CI metadata reports a revision; it
|
|
@@ -231,7 +236,7 @@ Actions artifact (or small committed Git files):
|
|
|
231
236
|
}
|
|
232
237
|
```
|
|
233
238
|
|
|
234
|
-
Paths are relative to the uploaded artifact root.
|
|
239
|
+
Paths are relative to the uploaded artifact root. Use your normal PR build;
|
|
235
240
|
for `actions/upload-artifact@v7`, use `archive: true`. Include companion files in
|
|
236
241
|
`paths` and explicitly map 1–32 command names to ordinary selected files. Gauge
|
|
237
242
|
validates Linux x86-64 ELF executables in its isolated artifact processor, restores
|
|
@@ -257,6 +262,49 @@ the source root. Gauge validates and extracts selected archives in isolation and
|
|
|
257
262
|
stages the Skill in the coding agent's native discovery location. File inputs use
|
|
258
263
|
`paths` to select committed files or files inside an Actions artifact.
|
|
259
264
|
|
|
265
|
+
Gauge tests the artifact uploaded by your PR's successful build. Keep your
|
|
266
|
+
existing checkout settings. The GitHub check links to the selected build;
|
|
267
|
+
rerun Gauge to test the latest build.
|
|
268
|
+
|
|
269
|
+
### Run from your own CI or local build
|
|
270
|
+
|
|
271
|
+
With CLI 0.18.0 or later, supply built inputs directly to run without the GitHub App:
|
|
272
|
+
|
|
273
|
+
```sh
|
|
274
|
+
gauge --org my-org evals run --input product-analytics=./dist/skill.zip --yes -o json > request.json
|
|
275
|
+
gauge --org my-org run-requests wait "$(jq -er '.runRequestId' request.json)" -o json > results.json
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
Set `GAUGE_API_TOKEN` as a CI secret and replace `my-org` with your workspace slug.
|
|
279
|
+
This can run immediately after your build in the same CI job.
|
|
280
|
+
|
|
281
|
+
Repeat `--input name=path` for each input. Skills accept a directory containing
|
|
282
|
+
`SKILL.md` or a ZIP/tar.gz archive; npm inputs accept a packed `.tgz`. Files and
|
|
283
|
+
binary inputs accept a directory, with configured paths relative to that directory.
|
|
284
|
+
Use a dedicated build directory: links are rejected, and common credential files
|
|
285
|
+
and `node_modules` are excluded. Each input is limited to 512 MiB and 20,000 files.
|
|
286
|
+
|
|
287
|
+
With `--input`, Gauge uses your current evaluation files, including uncommitted
|
|
288
|
+
edits. No Git checkout is required when all selected inputs are supplied locally.
|
|
289
|
+
Inputs without an override keep their configured source. You can omit `source`
|
|
290
|
+
in `gauge.json` to default to Git; the override replaces it for this run.
|
|
291
|
+
Website previews and private consumer repositories still require their configured
|
|
292
|
+
access. Public consumer repositories and the default workspace need no GitHub App.
|
|
293
|
+
|
|
294
|
+
`wait -o json` returns the verdict, session results, criterion explanations, and
|
|
295
|
+
session links. It exits nonzero for failed evaluations or execution errors and
|
|
296
|
+
still writes the results JSON. Automatic PR checks continue to use configured
|
|
297
|
+
sources; disable those checks in Gauge's GitHub settings if CI owns the trigger.
|
|
298
|
+
Save `results.json` as a CI artifact even when the check fails.
|
|
299
|
+
|
|
300
|
+
### Authenticated MCP servers
|
|
301
|
+
|
|
302
|
+
Connect your account in Gauge's MCP setup, completing OAuth in your browser when
|
|
303
|
+
required. Add the server's versioned reference to each case's `config.mcpRefs`.
|
|
304
|
+
Gauge manages credentials for the session; keep tokens out of prompts and Git.
|
|
305
|
+
|
|
306
|
+
### Website previews
|
|
307
|
+
|
|
260
308
|
Website inputs use `type: "website"` and a canonical HTTPS origin in `url`.
|
|
261
309
|
The preview can come from `source: { "type": "github-deployment", "environment": "preview" }`
|
|
262
310
|
or a `github-pr-comment` source with an exact `author`, literal `bodyIncludes`
|
|
@@ -268,8 +316,8 @@ They are fetched live rather than snapshotted. See the [preview source examples
|
|
|
268
316
|
limitations](../specs/docs-web-inputs.md), including Astro's existing bot comment.
|
|
269
317
|
|
|
270
318
|
Editor schemas are published at
|
|
271
|
-
`https://agents.withgauge.com/schemas/gauge/v2.
|
|
272
|
-
`https://agents.withgauge.com/schemas/gauge/v2.
|
|
319
|
+
`https://agents.withgauge.com/schemas/gauge/v2.7.0/gauge.json` and
|
|
320
|
+
`https://agents.withgauge.com/schemas/gauge/v2.7.0/case.json`. Associate them in editor
|
|
273
321
|
settings; `gauge.json` does not accept a `$schema` field. These release URLs are
|
|
274
322
|
immutable. Update earlier pilot configs by moving workflows into input sources,
|
|
275
323
|
replacing `use` with input `type` and `path`/`paths`, and moving cases into
|
|
@@ -343,75 +391,69 @@ funding by default, like Run Now in the web UI; `--bill-to-org` uses that org's
|
|
|
343
391
|
credits instead. Other users always use org credits and can launch only in orgs
|
|
344
392
|
they belong to.
|
|
345
393
|
|
|
346
|
-
## Optimizations
|
|
347
|
-
|
|
348
|
-
`gauge optimizations` improves one surface you own — a docs page or a skill —
|
|
349
|
-
against an eval or a preference prompt. Gauge measures a baseline, runs the
|
|
350
|
-
trials you author, judges them, and tells you which change won. It never writes
|
|
351
|
-
the change itself, and it never touches your repository. You edit on disk and
|
|
352
|
-
send the text; Gauge serves it to the sandboxed agent in place of the live page
|
|
353
|
-
or skill version.
|
|
394
|
+
## Optimizations
|
|
354
395
|
|
|
355
|
-
|
|
396
|
+
Start by finding runs that encountered the surface you want to improve:
|
|
356
397
|
|
|
357
398
|
```sh
|
|
358
|
-
gauge
|
|
359
|
-
gauge optimizations
|
|
360
|
-
gauge
|
|
361
|
-
|
|
362
|
-
gauge optimizations
|
|
363
|
-
$EDITOR start.md
|
|
364
|
-
gauge optimizations trials add cl_1 \
|
|
365
|
-
--title "Lead with the env var" --page https://docs.acme.dev/start \
|
|
366
|
-
--body-file start.md --launch --idempotency-key start-env-var --yes
|
|
367
|
-
|
|
399
|
+
gauge instructions optimization
|
|
400
|
+
gauge optimizations baselines --eval ev_1 -o json
|
|
401
|
+
gauge runs insight run_1 -o json
|
|
402
|
+
gauge runs logs run_1 -o json
|
|
403
|
+
gauge optimizations start --eval ev_1 --from-run run_1 --yes
|
|
368
404
|
gauge optimizations watch cl_1
|
|
369
|
-
gauge optimizations trial cl_1 R1 # score, verdict, sessions, errors
|
|
370
|
-
gauge optimizations change cl_1 R1 --body # the winning text, to paste
|
|
371
|
-
gauge optimizations adopt cl_1 R1 --yes # mark the winner; baseline stays fixed
|
|
372
|
-
gauge optimizations handoff cl_1 # instructions to land the winner in your source
|
|
373
|
-
gauge optimizations complete cl_1 --yes
|
|
374
405
|
```
|
|
375
406
|
|
|
376
|
-
`
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
407
|
+
`baselines` returns observed owned surfaces grouped by run, source evidence,
|
|
408
|
+
judging status, eligibility (including the baseline's complete run set), and a
|
|
409
|
+
`nextCursor` for `--cursor`. It uses the UI baseline picker's compatibility rules
|
|
410
|
+
and recent 100-batch eligibility window. Missing evidence is unknown; the list
|
|
411
|
+
is an analysis summary, not an exhaustive inventory of every trace event.
|
|
412
|
+
`--from-run` inherits the eligible baseline's harness/model and reuses its runs.
|
|
413
|
+
A stale choice fails instead of silently buying a new baseline. `--target` is
|
|
414
|
+
an alternative when you need to establish a baseline; do not combine it with
|
|
415
|
+
`--from-run`. Use `--preference` instead of `--eval` for preference prompts.
|
|
383
416
|
|
|
384
417
|
```sh
|
|
385
|
-
gauge optimizations authoring cl_1 > context.json
|
|
386
|
-
gauge optimizations
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
# "lever": "CLI", "intervention": {"kind": "CLI_INTERVENTION", "after": <bundle.json>}}}]}
|
|
390
|
-
gauge optimizations round preview cl_1 --file round.json # prints previewDigest
|
|
418
|
+
gauge optimizations authoring cl_1 > context.json
|
|
419
|
+
gauge optimizations page cl_1 --url https://docs.acme.dev/auth -o json
|
|
420
|
+
# Author round.json using context.revision and the observed baseline evidence.
|
|
421
|
+
gauge optimizations round preview cl_1 --file round.json -o json
|
|
391
422
|
gauge optimizations round save cl_1 --file round.json --digest <previewDigest>
|
|
392
|
-
gauge optimizations trials remove cl_1 R1.2 --yes # drop one draft
|
|
393
423
|
gauge optimizations launch cl_1 --yes
|
|
424
|
+
gauge optimizations watch cl_1
|
|
425
|
+
gauge optimizations trial cl_1 R1.1 -o json
|
|
426
|
+
gauge optimizations adopt cl_1 R1.1 --yes
|
|
427
|
+
gauge optimizations handoff cl_1
|
|
428
|
+
gauge optimizations complete cl_1 --yes
|
|
394
429
|
```
|
|
395
430
|
|
|
396
|
-
A
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
the
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
`
|
|
414
|
-
|
|
431
|
+
A round is `{revision, trials: [{trial: {title, hypothesis, lever, intervention}}]}`.
|
|
432
|
+
Docs `REWRITE_DOCS` proposals supply `after.pages` with URL and exact `edits`
|
|
433
|
+
(`find`/`replace`), or a complete `body` and `rewriteReason`. Check the page's
|
|
434
|
+
`source`: `scrape` means current fetched content, not the captured response.
|
|
435
|
+
|
|
436
|
+
For CLI trials, inspect `cliIntervention.guidance` and `targets` in authoring
|
|
437
|
+
and use `runs logs` for the cited command evidence. `cli-change --file change.json`
|
|
438
|
+
accepts v3 `{schemaVersion:3, target, rules, shippingIntent}` and returns a sealed
|
|
439
|
+
bundle to put in `CLI_INTERVENTION.after`. It does not save or launch. The old
|
|
440
|
+
catalog/invocation-ID workflow is retired. Skills use `skill-files` and the
|
|
441
|
+
complete edited file bundle.
|
|
442
|
+
|
|
443
|
+
A checkpoint is a saved state of a baseline run. Resuming it avoids repeating
|
|
444
|
+
earlier work. Gauge automatically selects a compatible checkpoint before the
|
|
445
|
+
affected interaction for docs and CLI candidates, or starts fresh with a reason.
|
|
446
|
+
You do not select a checkpoint; omit checkpoint choices and fork flags. Skills
|
|
447
|
+
start fresh. Inspect
|
|
448
|
+
`runs[].candidateExecution` for actual starting mode and `candidateDelivery`
|
|
449
|
+
for exposure, observed artifacts, and evidence completeness. A score improvement
|
|
450
|
+
does not prove the change was exercised. Report sample counts and failed sessions.
|
|
451
|
+
|
|
452
|
+
Read `nextAction` on optimization detail; `watch` exits 4 when input is needed.
|
|
453
|
+
Round save replaces pending drafts but never launches. Use `--yes` for approved
|
|
454
|
+
spending and `trials add --idempotency-key` when retrying single-trial writes.
|
|
455
|
+
Adoption selects a winner without changing the baseline or deploying the change;
|
|
456
|
+
`handoff` supplies instructions to implement it in your source repository.
|
|
415
457
|
|
|
416
458
|
## Models, reports, and scripting
|
|
417
459
|
|