@withgauge/cli 0.11.2 → 0.12.1

package/README.md

@@ -39,6 +39,83 @@ experiments,
 batches, brands, tags, actions, dashboards, members, org, provider keys,
 billing, usage, stats, query, and more.
 
+## Run committed Markdown evals
+
+File evals keep the task, run settings, and judged criteria in Git. Create
+`evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
+
+```markdown
+---
+criteria:
+  - name: Healthcheck responds
+    rubric: The endpoint returns HTTP 200 without authentication.
+config:
+  repoUrl: https://github.com/acme/app
+  repoRef: main
+  agents:
+    - agent: CODEX_CLI
+---
+Add a /health endpoint and run the relevant app checks.
+```
+
+Commit the file, then preview and run its exact committed bytes:
+
+```sh
+gauge evals plan --files evals/healthcheck.md
+gauge evals run --files evals/healthcheck.md --yes
+gauge run-requests wait <request-id> -o json
+```
+
+`--files` takes explicit paths; pass several paths after the flag for a
+multi-case request. The CLI reads `HEAD` through Git, so uncommitted edits are
+ignored and untracked cases fail. It hashes the committed case bytes and root
+config and submits their commit, digest, and repository origin. The server
+currently records that provenance as **submitted, unverified**; it does not
+fetch Git independently. `plan` resolves the configuration and estimates
+sessions without launching. Each case owns a `config` object with its own
+agent roster; supported fields are `agents`, `sampleCount`, `repoUrl`,
+`repoRef`, `profileId`, `skillRefs`, `mcpRefs`, `connectionIds`, and optional
+`addons.browser` (`session`, positive `projectId`, optional `region: us`).
+Skills and MCP assets use pinned `name@label` references. At least one agent
+and one criterion are required for each case. Omitted models use the agent's
+default model; omitted `sampleCount` runs one sample. With no `repoUrl`, the
+existing default scaffold is used.
+
+An optional root `gauge.json` selects the organization and one product build:
+
+```json
+{"org":"my-org","product":{"type":"npm-cli"}}
+```
+
+For an npm CLI, Gauge uses the committed package manifest and lockfile, runs
+its `build` script when present, packs and installs the package with production
+dependencies, and exposes its declared CLI commands with the pinned Node
+runtime. Set `product.build` to a different build command when needed.
+The source install is frozen; Gauge resolves a fresh production install once per
+run request, archives that installed tree for every case, and records its
+runtime lockfile digest in the build receipt. New requests can resolve newer
+transitive dependencies until the product pins them.
+Each new CLI launch currently rebuilds, even at the same source SHA; bundles
+are shared across cases and reused after completed preparation within a request,
+but there is no cache across requests yet.
+The first adapter requires the pinned pnpm 10.34.5 package manager and a
+`pnpm-lock.yaml`.
+`--org` overrides the committed `org`; otherwise the CLI falls back to your
+configured default organization.
+
+For other products, `{"prepare":"pnpm run gauge:prepare","install":"./install.sh"}`
+remains available. `prepare` runs once at the exact definition commit with
+`GAUGE_OUTPUT_DIR` pointing to an empty directory; `install` is an optional
+bundle-relative script run once per consumer session. A committed
+`.gauge/prepare.sh` enables that preparation automatically when `prepare` is
+absent. `product` cannot be combined with either script form. The output is
+attached to every case. There is no authored resource manifest or per-case
+`inputs` field. Project-level run defaults are rejected. The optional
+`version` defaults to 1. Preparation requires a public HTTPS Git origin.
+Saved evals remain available with `gauge evals run <id>`.
+As with saved evals, superusers use platform funding by default and can add
+`--bill-to-org` to charge the selected organization.
+
 ## Measurements own their configuration
 
 Every runnable thing carries its own complete run configuration — repository,
@@ -138,6 +215,24 @@ Edit a skill with `--edit-skill <versionId> --skill-dir <path>`: the directory
 holds the complete bundle, `SKILL.md` included. `trials template --kind <kind>`
 prints a skeleton for the `--file` form.
 
+A chat-guided round, such as a CLI change, goes through preview and save. Save
+replaces the pending drafts, so it can fix a draft that is wrong:
+
+```sh
+gauge optimizations authoring cl_1 > context.json   # revision, CLI targets, invocations
+gauge optimizations invocation cl_1 inv_1           # one recorded baseline invocation
+gauge optimizations cli-change cl_1 --file change.json > bundle.json
+# round.json: {"revision": "...", "trials": [{"trial": {"title": "...",
+#   "lever": "CLI", "intervention": {"kind": "CLI_INTERVENTION", "after": <bundle.json>}}}]}
+gauge optimizations round preview cl_1 --file round.json   # prints previewDigest
+gauge optimizations round save cl_1 --file round.json --digest <previewDigest>
+gauge optimizations trials remove cl_1 R1.2 --yes    # drop one draft
+gauge optimizations launch cl_1 --yes
+```
+
+A `completeOutput` literal matcher is an exact, case-sensitive match. Copy it
+from the recorded command output.
+
 Every detail carries `nextAction` — `START`, `WAIT`, `AUTHOR_TRIAL`,
 `LAUNCH_ROUND`, `ADOPT`, `RESUME`, or `NONE`. Read it (`get -o json`) instead of
 inferring whose turn it is from round and trial statuses. `watch` exits `4` when