@withgauge/cli 0.11.0 → 0.12.0

package/README.md

@@ -39,6 +39,83 @@ experiments,
 batches, brands, tags, actions, dashboards, members, org, provider keys,
 billing, usage, stats, query, and more.
 
+## Run committed Markdown evals
+
+File evals keep the task, run settings, and judged criteria in Git. Create
+`evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
+
+```markdown
+---
+criteria:
+  - name: Healthcheck responds
+    rubric: The endpoint returns HTTP 200 without authentication.
+config:
+  repoUrl: https://github.com/acme/app
+  repoRef: main
+  agents:
+    - agent: CODEX_CLI
+---
+Add a /health endpoint and run the relevant app checks.
+```
+
+Commit the file, then preview and run its exact committed bytes:
+
+```sh
+gauge evals plan --files evals/healthcheck.md
+gauge evals run --files evals/healthcheck.md --yes
+gauge run-requests wait <request-id> -o json
+```
+
+`--files` takes explicit paths; pass several paths after the flag for a
+multi-case request. The CLI reads `HEAD` through Git, so uncommitted edits are
+ignored and untracked cases fail. It hashes the committed case bytes and root
+config and submits their commit, digest, and repository origin. The server
+currently records that provenance as **submitted, unverified**; it does not
+fetch Git independently. `plan` resolves the configuration and estimates
+sessions without launching. Each case owns a `config` object with its own
+agent roster; supported fields are `agents`, `sampleCount`, `repoUrl`,
+`repoRef`, `profileId`, `skillRefs`, `mcpRefs`, `connectionIds`, and optional
+`addons.browser` (`session`, positive `projectId`, optional `region: us`).
+Skills and MCP assets use pinned `name@label` references. At least one agent
+and one criterion are required for each case. Omitted models use the agent's
+default model; omitted `sampleCount` runs one sample. With no `repoUrl`, the
+existing default scaffold is used.
+
+An optional root `gauge.json` selects the organization and one product build:
+
+```json
+{"org":"my-org","product":{"type":"npm-cli"}}
+```
+
+For an npm CLI, Gauge uses the committed package manifest and lockfile, runs
+its `build` script when present, packs and installs the package with production
+dependencies, and exposes its declared CLI commands with the pinned Node
+runtime. Set `product.build` to a different build command when needed.
+The source install is frozen; Gauge resolves a fresh production install once per
+run request, archives that installed tree for every case, and records its
+runtime lockfile digest in the build receipt. New requests can resolve newer
+transitive dependencies until the product pins them.
+Each new CLI launch currently rebuilds, even at the same source SHA; bundles
+are shared across cases and reused after completed preparation within a request,
+but there is no cache across requests yet.
+The first adapter requires the pinned pnpm 10.34.5 package manager and a
+`pnpm-lock.yaml`.
+`--org` overrides the committed `org`; otherwise the CLI falls back to your
+configured default organization.
+
+For other products, `{"prepare":"pnpm run gauge:prepare","install":"./install.sh"}`
+remains available. `prepare` runs once at the exact definition commit with
+`GAUGE_OUTPUT_DIR` pointing to an empty directory; `install` is an optional
+bundle-relative script run once per consumer session. A committed
+`.gauge/prepare.sh` enables that preparation automatically when `prepare` is
+absent. `product` cannot be combined with either script form. The output is
+attached to every case. There is no authored resource manifest or per-case
+`inputs` field. Project-level run defaults are rejected. The optional
+`version` defaults to 1. Preparation requires a public HTTPS Git origin.
+Saved evals remain available with `gauge evals run <id>`.
+As with saved evals, superusers use platform funding by default and can add
+`--bill-to-org` to charge the selected organization.
+
 ## Measurements own their configuration
 
 Every runnable thing carries its own complete run configuration — repository,
@@ -98,6 +175,10 @@ immediate launches. They use the stored configuration and ask before spending
 (`--yes` skips the prompt for scripts).
 `gauge preference run <id> --scenario <scenarioId>` (repeatable) runs a subset;
 omit it and every scenario runs. Neither launch touches the cadence clock.
+Superusers can use `--org <slug>` to launch in any org. Their runs use PLATFORM
+funding by default, like Run Now in the web UI; `--bill-to-org` uses that org's
+credits instead. Other users always use org credits and can launch only in orgs
+they belong to.
 
 ## Optimizations: Gauge measures, you author
 
@@ -125,7 +206,7 @@ gauge optimizations trials add cl_1 \
 gauge optimizations watch cl_1
 gauge optimizations trial cl_1 R1                   # score, verdict, sessions
 gauge optimizations change cl_1 R1 --body           # the winning text, to paste
-gauge optimizations adopt cl_1 R1 --yes             # the new moving baseline
+gauge optimizations adopt cl_1 R1 --yes             # mark the winner; baseline stays fixed
 gauge optimizations complete cl_1 --yes
 ```