@withgauge/cli 0.17.1 → 0.19.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +114 -72
  2. package/dist/index.js +2092 -1597
  3. package/package.json +2 -1
package/README.md CHANGED
@@ -76,7 +76,7 @@ config:
76
76
  Add a /health endpoint and run the relevant app checks.
77
77
  ```
78
78
 
79
- Check your working-tree setup, then commit the files before planning and running:
79
+ Check your setup, then commit the files before planning and running with Git inputs:
80
80
 
81
81
  ```sh
82
82
  gauge evals verify # offline; includes uncommitted working-tree edits
@@ -87,7 +87,8 @@ gauge run-requests wait <request-id> -o json
87
87
  ```
88
88
 
89
89
  Cases default to `gauge-evals/**/*.md`. `--files` accepts quoted paths/globs and overrides
90
- that convention for `plan` and `run`, which read committed Git HEAD, ignoring working-tree edits.
90
+ that convention for `plan` and `run`. By default, these commands read committed Git
91
+ HEAD. With `--input`, they read your current files, including uncommitted edits.
91
92
  Each case owns its agents, models, consumer repository/ref, Persona and criteria.
92
93
  The optional `config.inputs` selects named inputs; all attach by default.
93
94
 
@@ -104,7 +105,7 @@ event filters and runtime conditions are not evaluated against a candidate PR.
104
105
  Live input delivery, artifact contents, sandbox execution, App activation, and
105
106
  account settings are outside this command's scope and do not create unknown rows.
106
107
  Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
107
- No workflow code is executed. Commit files before `plan` or `run` consumes them.
108
+ No workflow code is executed.
108
109
 
109
110
  `gauge evals verify -o json` writes exactly one JSON object to stdout, including on
110
111
  validation failure. The versioned report provides:
@@ -150,15 +151,19 @@ alongside your product:
150
151
  }
151
152
  ```
152
153
 
153
- Gauge waits for the existing workflow at the pushed revision and consumes its
154
- artifacts by ID. Omit `source.artifact` to select a sole artifact, or use a name
155
- pattern when the workflow uploads several. It never triggers or performs a build. Connect the Gauge
156
- GitHub App with repository access and Actions read permission. Plan previews CI
157
- readiness; waiting requests create no sessions and incur no session debit.
158
- Each input names its workflow. Gauge captures one successful run/attempt per
159
- workflow before downloading any input. A committed org and matching GitHub App
160
- installation enable automatic same-repository PR evaluations for repositories and
161
- target branches enabled in Settings → GitHub (initially enabled for `main`).
154
+ Gauge waits for your PR build and tests its uploaded artifacts. Keep the workflow's
155
+ existing checkout settings, including GitHub's default merge checkout. Omit
156
+ `source.artifact` when the workflow uploads one artifact, or select it by name.
157
+
158
+ For automatic checks, connect the Gauge GitHub App with repository access and
159
+ Actions read permission. Set `org` and enable the repository and target branches
160
+ in **Settings → GitHub**. Waiting for inputs starts no sessions and spends no
161
+ session credits.
162
+ Fork pull requests are disabled by default. Check **Allow fork pull requests** to let
163
+ workspace members authorize an evaluation from its GitHub check. Authorization
164
+ covers one PR revision and uses workspace credits; new revisions and explicit
165
+ reruns require fresh approval. Configured inputs determine when an authorized
166
+ evaluation is ready to run.
162
167
  A failed run or absent/expired artifact returns an actionable input failure.
163
168
  Absent runs have a five-minute discovery grace period; known running or queued
164
169
  CI can continue past an hour. Abandoned requests expire after seven days. Reruns retain captured bytes. CI metadata reports a revision; it
@@ -231,7 +236,7 @@ Actions artifact (or small committed Git files):
231
236
  }
232
237
  ```
233
238
 
234
- Paths are relative to the uploaded artifact root. Build the exact candidate SHA;
239
+ Paths are relative to the uploaded artifact root. Use your normal PR build;
235
240
  for `actions/upload-artifact@v7`, use `archive: true`. Include companion files in
236
241
  `paths` and explicitly map 1–32 command names to ordinary selected files. Gauge
237
242
  validates Linux x86-64 ELF executables in its isolated artifact processor, restores
@@ -257,6 +262,49 @@ the source root. Gauge validates and extracts selected archives in isolation and
257
262
  stages the Skill in the coding agent's native discovery location. File inputs use
258
263
  `paths` to select committed files or files inside an Actions artifact.
259
264
 
265
+ Gauge tests the artifact uploaded by your PR's successful build. Keep your
266
+ existing checkout settings. The GitHub check links to the selected build;
267
+ rerun Gauge to test the latest build.
268
+
269
+ ### Run from your own CI or local build
270
+
271
+ With CLI 0.18.0 or later, supply built inputs directly to run without the GitHub App:
272
+
273
+ ```sh
274
+ gauge --org my-org evals run --input product-analytics=./dist/skill.zip --yes -o json > request.json
275
+ gauge --org my-org run-requests wait "$(jq -er '.runRequestId' request.json)" -o json > results.json
276
+ ```
277
+
278
+ Set `GAUGE_API_TOKEN` as a CI secret and replace `my-org` with your workspace slug.
279
+ This can run immediately after your build in the same CI job.
280
+
281
+ Repeat `--input name=path` for each input. Skills accept a directory containing
282
+ `SKILL.md` or a ZIP/tar.gz archive; npm inputs accept a packed `.tgz`. Files and
283
+ binary inputs accept a directory, with configured paths relative to that directory.
284
+ Use a dedicated build directory: links are rejected, and common credential files
285
+ and `node_modules` are excluded. Each input is limited to 512 MiB and 20,000 files.
286
+
287
+ With `--input`, Gauge uses your current evaluation files, including uncommitted
288
+ edits. No Git checkout is required when all selected inputs are supplied locally.
289
+ Inputs without an override keep their configured source. You can omit `source`
290
+ in `gauge.json` to default to Git; the override replaces it for this run.
291
+ Website previews and private consumer repositories still require their configured
292
+ access. Public consumer repositories and the default workspace need no GitHub App.
293
+
294
+ `wait -o json` returns the verdict, session results, criterion explanations, and
295
+ session links. It exits nonzero for failed evaluations or execution errors and
296
+ still writes the results JSON. Automatic PR checks continue to use configured
297
+ sources; disable those checks in Gauge's GitHub settings if CI owns the trigger.
298
+ Save `results.json` as a CI artifact even when the check fails.
299
+
300
+ ### Authenticated MCP servers
301
+
302
+ Connect your account in Gauge's MCP setup, completing OAuth in your browser when
303
+ required. Add the server's versioned reference to each case's `config.mcpRefs`.
304
+ Gauge manages credentials for the session; keep tokens out of prompts and Git.
305
+
306
+ ### Website previews
307
+
260
308
  Website inputs use `type: "website"` and a canonical HTTPS origin in `url`.
261
309
  The preview can come from `source: { "type": "github-deployment", "environment": "preview" }`
262
310
  or a `github-pr-comment` source with an exact `author`, literal `bodyIncludes`
@@ -268,8 +316,8 @@ They are fetched live rather than snapshotted. See the [preview source examples
268
316
  limitations](../specs/docs-web-inputs.md), including Astro's existing bot comment.
269
317
 
270
318
  Editor schemas are published at
271
- `https://agents.withgauge.com/schemas/gauge/v2.6.0/gauge.json` and
272
- `https://agents.withgauge.com/schemas/gauge/v2.6.0/case.json`. Associate them in editor
319
+ `https://agents.withgauge.com/schemas/gauge/v2.7.0/gauge.json` and
320
+ `https://agents.withgauge.com/schemas/gauge/v2.7.0/case.json`. Associate them in editor
273
321
  settings; `gauge.json` does not accept a `$schema` field. These release URLs are
274
322
  immutable. Update earlier pilot configs by moving workflows into input sources,
275
323
  replacing `use` with input `type` and `path`/`paths`, and moving cases into
@@ -343,75 +391,69 @@ funding by default, like Run Now in the web UI; `--bill-to-org` uses that org's
343
391
  credits instead. Other users always use org credits and can launch only in orgs
344
392
  they belong to.
345
393
 
346
- ## Optimizations: Gauge measures, you author
347
-
348
- `gauge optimizations` improves one surface you own — a docs page or a skill —
349
- against an eval or a preference prompt. Gauge measures a baseline, runs the
350
- trials you author, judges them, and tells you which change won. It never writes
351
- the change itself, and it never touches your repository. You edit on disk and
352
- send the text; Gauge serves it to the sandboxed agent in place of the live page
353
- or skill version.
394
+ ## Optimizations
354
395
 
355
- A scripted or agent-driven loop:
396
+ Start by finding runs that encountered the surface you want to improve:
356
397
 
357
398
  ```sh
358
- gauge optimizations diagnose --eval es_1 # where the sessions fail
359
- gauge optimizations start --eval es_1 --target claude-code:claude-opus-4-8 --yes
360
- gauge optimizations watch cl_1 # exit 4 = your turn
361
-
362
- gauge optimizations page cl_1 --url https://docs.acme.dev/start > start.md
363
- $EDITOR start.md
364
- gauge optimizations trials add cl_1 \
365
- --title "Lead with the env var" --page https://docs.acme.dev/start \
366
- --body-file start.md --launch --idempotency-key start-env-var --yes
367
-
399
+ gauge instructions optimization
400
+ gauge optimizations baselines --eval ev_1 -o json
401
+ gauge runs insight run_1 -o json
402
+ gauge runs logs run_1 -o json
403
+ gauge optimizations start --eval ev_1 --from-run run_1 --yes
368
404
  gauge optimizations watch cl_1
369
- gauge optimizations trial cl_1 R1 # score, verdict, sessions, errors
370
- gauge optimizations change cl_1 R1 --body # the winning text, to paste
371
- gauge optimizations adopt cl_1 R1 --yes # mark the winner; baseline stays fixed
372
- gauge optimizations handoff cl_1 # instructions to land the winner in your source
373
- gauge optimizations complete cl_1 --yes
374
405
  ```
375
406
 
376
- `page` and `skill-files` print the exact frozen content a trial starts from.
377
- Edit a skill with `--edit-skill <versionId> --skill-dir <path>`: the directory
378
- holds the complete bundle, `SKILL.md` included. `trials template --kind <kind>`
379
- prints a skeleton for the `--file` form.
380
-
381
- A chat-guided round, such as a CLI change, goes through preview and save. Save
382
- replaces the pending drafts, so it can fix a draft that is wrong:
407
+ `baselines` returns observed owned surfaces grouped by run, source evidence,
408
+ judging status, eligibility (including the baseline's complete run set), and a
409
+ `nextCursor` for `--cursor`. It uses the UI baseline picker's compatibility rules
410
+ and recent 100-batch eligibility window. Missing evidence is unknown; the list
411
+ is an analysis summary, not an exhaustive inventory of every trace event.
412
+ `--from-run` inherits the eligible baseline's harness/model and reuses its runs.
413
+ A stale choice fails instead of silently buying a new baseline. `--target` is
414
+ an alternative when you need to establish a baseline; do not combine it with
415
+ `--from-run`. Use `--preference` instead of `--eval` for preference prompts.
383
416
 
384
417
  ```sh
385
- gauge optimizations authoring cl_1 > context.json # revision, CLI targets, invocations
386
- gauge optimizations invocation cl_1 inv_1 # one recorded baseline invocation
387
- gauge optimizations cli-change cl_1 --file change.json > bundle.json
388
- # round.json: {"revision": "...", "trials": [{"trial": {"title": "...",
389
- # "lever": "CLI", "intervention": {"kind": "CLI_INTERVENTION", "after": <bundle.json>}}}]}
390
- gauge optimizations round preview cl_1 --file round.json # prints previewDigest
418
+ gauge optimizations authoring cl_1 > context.json
419
+ gauge optimizations page cl_1 --url https://docs.acme.dev/auth -o json
420
+ # Author round.json using context.revision and the observed baseline evidence.
421
+ gauge optimizations round preview cl_1 --file round.json -o json
391
422
  gauge optimizations round save cl_1 --file round.json --digest <previewDigest>
392
- gauge optimizations trials remove cl_1 R1.2 --yes # drop one draft
393
423
  gauge optimizations launch cl_1 --yes
424
+ gauge optimizations watch cl_1
425
+ gauge optimizations trial cl_1 R1.1 -o json
426
+ gauge optimizations adopt cl_1 R1.1 --yes
427
+ gauge optimizations handoff cl_1
428
+ gauge optimizations complete cl_1 --yes
394
429
  ```
395
430
 
396
- A `completeOutput` literal matcher is an exact, case-sensitive match. Copy it
397
- from the recorded command output.
398
-
399
- Every detail carries `nextAction` — `START`, `WAIT`, `AUTHOR_TRIAL`,
400
- `LAUNCH_ROUND`, `ADOPT`, `RESUME`, or `NONE`. Read it (`get -o json`) instead of
401
- inferring whose turn it is from round and trial statuses. `watch` exits `4` when
402
- the answer is you.
403
-
404
- `--idempotency-key` makes `trials add` safe to retry: the same key returns the
405
- trial the first call created, and never buys a second set of sessions.
406
-
407
- There is no round limit or credit budget. Each round waits for you; `complete`
408
- ends the optimization. Adoption marks the winner and leaves the baseline fixed.
409
- It does not open a pull request. `handoff` prints the same instructions as the
410
- web app's "Copy instructions for agent"; follow them in your own source.
411
-
412
- A failed or timed-out session drops out of the score. `trial`, `get`, and
413
- `watch` print a `WARNING` line for each one, with its failure reason; in JSON,
414
- read `failedSessions` on the trial and `failureReason` on each run.
431
+ A round is `{revision, trials: [{trial: {title, hypothesis, lever, intervention}}]}`.
432
+ Docs `REWRITE_DOCS` proposals supply `after.pages` with URL and exact `edits`
433
+ (`find`/`replace`), or a complete `body` and `rewriteReason`. Check the page's
434
+ `source`: `scrape` means current fetched content, not the captured response.
435
+
436
+ For CLI trials, inspect `cliIntervention.guidance` and `targets` in authoring
437
+ and use `runs logs` for the cited command evidence. `cli-change --file change.json`
438
+ accepts v3 `{schemaVersion:3, target, rules, shippingIntent}` and returns a sealed
439
+ bundle to put in `CLI_INTERVENTION.after`. It does not save or launch. The old
440
+ catalog/invocation-ID workflow is retired. Skills use `skill-files` and the
441
+ complete edited file bundle.
442
+
443
+ A checkpoint is a saved state of a baseline run. Resuming it avoids repeating
444
+ earlier work. Gauge automatically selects a compatible checkpoint before the
445
+ affected interaction for docs and CLI candidates, or starts fresh with a reason.
446
+ You do not select a checkpoint; omit checkpoint choices and fork flags. Skills
447
+ start fresh. Inspect
448
+ `runs[].candidateExecution` for actual starting mode and `candidateDelivery`
449
+ for exposure, observed artifacts, and evidence completeness. A score improvement
450
+ does not prove the change was exercised. Report sample counts and failed sessions.
451
+
452
+ Read `nextAction` on optimization detail; `watch` exits 4 when input is needed.
453
+ Round save replaces pending drafts but never launches. Use `--yes` for approved
454
+ spending and `trials add --idempotency-key` when retrying single-trial writes.
455
+ Adoption selects a winner without changing the baseline or deploying the change;
456
+ `handoff` supplies instructions to implement it in your source repository.
415
457
 
416
458
  ## Models, reports, and scripting
417
459