@withgauge/cli 0.14.0 → 0.15.0

package/README.md

@@ -49,6 +49,8 @@ never drifts from the commands it describes.
 
 Use `gauge instructions evals` before creating evals: it covers developer tasks,
 model selection, observable judging criteria, and a complete migration example.
+Use `gauge instructions preference` to investigate product choices, source
+exposure, and interventions, from individual traces to larger cohorts.
 Use `gauge instructions optimization` for the full optimization workflow.
 `gauge instructions --list` shows available topics.
 `gauge instructions evals --install` installs the eval guidance as a local skill.
@@ -61,7 +63,7 @@ billing, usage, stats, query, and more.
 ## Run committed Markdown evals
 
 File evals keep the task, run settings, and judged criteria in Git. Create
-`evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
+`gauge-evals/healthcheck.md` with YAML frontmatter and a Markdown task body:
 
 ```markdown
 ---
@@ -74,66 +76,194 @@ config:
   agents:
     - agent: CODEX_CLI
 ---
+
 Add a /health endpoint and run the relevant app checks.
 ```
 
-Commit the file, then preview and run its exact committed bytes:
+Check your working-tree setup, then commit the files before planning and running:
 
 ```sh
-gauge evals plan --files evals/healthcheck.md
-gauge evals run --files evals/healthcheck.md --yes
+gauge evals verify              # offline; includes uncommitted working-tree edits
+gauge evals verify -o json      # versioned global/per-input yes/no/unknown checklist
+gauge evals plan
+gauge evals run --yes
 gauge run-requests wait <request-id> -o json
 ```
 
-`--files` takes explicit paths; pass several paths after the flag for a
-multi-case request. The CLI reads `HEAD` through Git, so uncommitted edits are
-ignored and untracked cases fail. It hashes the committed case bytes and root
-config and submits their commit, digest, and repository origin. The server
-currently records that provenance as **submitted, unverified**; it does not
-fetch Git independently. `plan` resolves the configuration and estimates
-sessions without launching. Each case owns a `config` object with its own
-agent roster; supported fields are `agents`, `sampleCount`, `repoUrl`,
-`repoRef`, `profileId`, `skillRefs`, `mcpRefs`, `connectionIds`, and optional
-`addons.browser` (`session`, positive `projectId`, optional `region: us`).
-Skills and MCP assets use pinned `name@label` references. At least one agent
-and one criterion are required for each case. Omitted models use the agent's
-default model; omitted `sampleCount` runs one sample. With no `repoUrl`, the
-existing default scaffold is used.
-
-An optional root `gauge.json` selects the organization and one product build:
+Cases default to `gauge-evals/**/*.md`. `--files` accepts quoted paths/globs and overrides
+that convention for `plan` and `run`, which read committed Git HEAD, ignoring working-tree edits.
+Each case owns its agents, models, consumer repository/ref, Persona and criteria.
+The optional `config.inputs` selects named inputs; all attach by default.
+
+`verify` reads the repository's current files, including non-ignored untracked files,
+without login, network access, a PR, built CI artifacts, or sandbox compute. It checks
+configuration, case input references, Git input paths, and recognizable workflow
+checkout/upload declarations. An input-only repository with no cases is valid setup.
+Each attempted global/per-input check reports `yes` (established locally), `no`
+(a definite problem), or `unknown` (a specific static question could not be resolved,
+such as an expression-based artifact name or a referenced reusable workflow).
+Missing optional configuration and zero cases are informational findings, excluded
+from checklist counts. Eligible workflow triggers are checked as declarations;
+event filters and runtime conditions are not evaluated against a candidate PR.
+Live input delivery, artifact contents, sandbox execution, App activation, and
+account settings are outside this command's scope and do not create unknown rows.
+Exit 1 means at least one `no`; exit 0 means no definite static problem was found.
+No workflow code is executed. Commit files before `plan` or `run` consumes them.
+
+`gauge evals verify -o json` writes exactly one JSON object to stdout, including on
+validation failure. The versioned report provides:
+
+- `version: 1` and `mode: "local-static"` identify the report contract and scope.
+- `status` is `failed` if any check is `no`, otherwise `incomplete` if any check is
+  `unknown`, otherwise `passed`. It describes static checks only.
+- `summary` contains `yes`, `no`, and `unknown` counts.
+- `global` and `inputs[].checks` contain checks with stable `id`, `state`, and
+  explanatory `detail` fields; `inputs[].name` identifies each input.
+- `facts.configFilePresent` and `facts.caseCount` expose discovered facts without
+  parsing prose. A `null` value means inspection did not reach that fact; zero cases
+  is distinct from case discovery being skipped or failing.
+- `info` contains informational findings with stable `id` and `detail` fields.
+- `notChecked` lists excluded capabilities as machine-readable identifiers.
+
+For example, an agent can select unresolved checks with:
+
+```sh
+gauge evals verify -o json | jq '{status, facts, global: [.global[] | select(.state != "yes")], inputs: [.inputs[] | {name, checks: [.checks[] | select(.state != "yes")]}]}'
+```
+
+An optional `gauge.json` declares inputs produced by your normal CI or committed
+alongside your product:
+
+```json
+{
+  "version": 2,
+  "org": "my-org",
+  "inputs": {
+    "cli": {
+      "type": "npm-package",
+      "source": { "type": "github-actions", "workflow": ".github/workflows/package.yml" },
+      "path": "*.tgz",
+      "install": "environment"
+    },
+    "docs": {
+      "type": "files",
+      "source": { "type": "git" },
+      "paths": ["docs/**/*.md"]
+    }
+  }
+}
+```
+
+Gauge waits for the existing workflow at the pushed revision and consumes its
+artifacts by ID. Omit `source.artifact` to select a sole artifact, or use a name
+pattern when the workflow uploads several. It never triggers or performs a build. Connect the Gauge
+GitHub App with repository access and Actions read permission. Plan previews CI
+readiness; waiting requests create no sessions and incur no session debit.
+Each input names its workflow. Gauge captures one successful run/attempt per
+workflow before downloading any input. A committed org and matching GitHub App
+installation enable automatic same-repository PR evaluations for repositories and
+target branches enabled in Settings → GitHub (initially enabled for `main`).
+A failed run or absent/expired artifact returns an actionable input failure.
+Absent runs have a five-minute discovery grace period; known running or queued
+CI can continue past an hour. Abandoned requests expire after seven days. Reruns retain captured bytes. CI metadata reports a revision; it
+does not independently prove which checkout the workflow compiled.
+
+To run automatic PR checks only when relevant files change, add `checks` to
+`gauge.json`:
 
 ```json
-{"org":"my-org","product":{"type":"npm-cli"}}
+"checks": {
+  "paths": ["agents/**", "docs.json", "snippets/**"]
+}
 ```
 
-For an npm CLI, Gauge uses the committed package manifest and lockfile, runs
-its `build` script when present, packs and installs the package with production
-dependencies, and exposes its declared CLI commands with the pinned Node
-runtime. Set `product.build` to a different build command when needed.
-The source install is frozen; Gauge resolves a fresh production install once per
-run request, archives that installed tree for every case, and records its
-runtime lockfile digest in the build receipt. New requests can resolve newer
-transitive dependencies until the product pins them.
-Each new CLI launch currently rebuilds, even at the same source SHA; bundles
-are shared across cases and reused after completed preparation within a request,
-but there is no cache across requests yet.
-The first adapter requires the pinned pnpm 10.34.5 package manager and a
-`pnpm-lock.yaml`.
-`--org` overrides the committed `org`; otherwise the CLI falls back to your
-configured default organization.
-
-For other products, `{"prepare":"pnpm run gauge:prepare","install":"./install.sh"}`
-remains available. `prepare` runs once at the exact definition commit with
-`GAUGE_OUTPUT_DIR` pointing to an empty directory; `install` is an optional
-bundle-relative script run once per consumer session. A committed
-`.gauge/prepare.sh` enables that preparation automatically when `prepare` is
-absent. `product` cannot be combined with either script form. The output is
-attached to every case. There is no authored resource manifest or per-case
-`inputs` field. Project-level run defaults are rejected. The optional
-`version` defaults to 1. Preparation requires a public HTTPS Git origin.
-Saved evals remain available with `gauge evals run <id>`.
-As with saved evals, superusers use platform funding by default and can add
-`--bill-to-org` to charge the selected organization.
+Paths are repository-relative, case-sensitive globs; matching any pattern runs
+the checks. `**` matches nested directories, including hidden files. Use 1–32
+patterns; exclusions (`!`) are not supported. Changes to `gauge.json` or your
+configured case paths always count. Deleted files and both names of renamed
+files are included. Without `checks`, every eligible PR update runs as before.
+
+The first check uses the PR diff. After a passing evaluation, later updates
+compare against that evaluated commit, so unrelated follow-up edits can skip.
+A failed or unfinished evaluation, a changed target branch, or an unavailable
+or incomplete diff causes a run. Skipped checks start no sessions and reserve
+no credits. GitHub's rerun action runs regardless of paths; explicit CLI runs
+also ignore this automatic-check filter.
+
+File inputs stay outside the consumer repository and their paths are provided to
+the agent. npm packages install with ordinary npm using the VM's Node/npm runtime.
+`install` defaults to `environment`: a separate tools directory with the package's
+own declared commands on PATH, including login shells. `workspace` adds a consumer
+application dependency and accepts libraries without commands. Transitive registry
+dependency ranges remain live. Plan output shows the installation destination and,
+when a previous matching preparation captured it, command metadata. Otherwise it
+reports that commands will be determined during preparation; planning never downloads
+archives or starts VMs.
+
+Native executables use `type: "binary"`, with already unpacked files in a GitHub
+Actions artifact (or small committed Git files):
+
+```json
+"gh": {
+  "type": "binary",
+  "source": {
+    "type": "github-actions",
+    "workflow": ".github/workflows/package.yml",
+    "artifact": "gh-linux"
+  },
+  "paths": ["dist/**"],
+  "platform": "linux-x64",
+  "commands": { "gh": "dist/bin/gh" }
+}
+```
+
+Paths are relative to the uploaded artifact root. Build the exact candidate SHA;
+for `actions/upload-artifact@v7`, use `archive: true`. Include companion files in
+`paths` and explicitly map 1–32 command names to ordinary selected files. Gauge
+validates Linux x86-64 ELF executables in its isolated artifact processor, restores
+executable permissions, preserves the bundle outside the consumer repository, and
+exposes the commands on PATH, including login shells. A candidate command overrides
+an installed product command with the same name; two inputs selected by the same
+case cannot expose the same command (including captured npm environment commands).
+
+The first version supports ELF executables and PIE binaries compatible with the
+consumer Linux runtime. It does not unpack nested release archives, install OS
+packages, alter library search paths, or run arbitrary installation/version-check
+commands. Prefer a static binary or build against a compatible Linux runtime;
+header validation cannot prove shared-library compatibility. Scripts, macOS,
+Windows, and ARM executables are unsupported. A missing loader/library encountered
+when the agent invokes the command appears in session execution evidence, not in
+static validation. Harness/runtime names (including shells, Node/npm, Python, Git,
+archive/setup utilities, and agent commands) are reserved; use a different alias
+for such a candidate. `GAUGE_INPUTS` gives the preserved bundle path.
+
+A Skill input has `type: "skill"` and `path: "skills/product"`, or a path to a ZIP,
+`.tar.gz`, or `.tgz` whose root contains `SKILL.md`. Use `path: "."` for a Skill at
+the source root. Gauge validates and extracts selected archives in isolation and
+stages the Skill in the coding agent's native discovery location. File inputs use
+`paths` to select committed files or files inside an Actions artifact.
+
+Website inputs use `type: "website"` and a canonical HTTPS origin in `url`.
+The preview can come from `source: { "type": "github-deployment", "environment": "preview" }`
+or a `github-pr-comment` source with an exact `author`, literal `bodyIncludes`
+marker, and a readiness `pattern` with named `url` and `sha` captures. Gauge selects
+candidate-specific evidence; timestamps alone cannot establish the revision.
+Comments must belong to a unique open same-repository PR at the candidate head.
+Previews must be public HTTPS origins, without authentication or path prefixes.
+They are fetched live rather than snapshotted. See the [preview source examples and
+limitations](../specs/docs-web-inputs.md), including Astro's existing bot comment.
+
+Editor schemas are published at
+`https://agents.withgauge.com/schemas/gauge/v2.4.0/gauge.json` and
+`https://agents.withgauge.com/schemas/gauge/v2.4.0/case.json`. Associate them in editor
+settings; `gauge.json` does not accept a `$schema` field. These release URLs are
+immutable. Update earlier pilot configs by moving workflows into input sources,
+replacing `use` with input `type` and `path`/`paths`, and moving cases into
+`gauge-evals/` (or setting `cases` explicitly).
+
+`--org` overrides the committed organization. Root `cases` can change discovery;
+root `version` defaults to 2. Saved evals use `gauge evals run <id>`. Superusers
+use platform funding by default and can add `--bill-to-org`.
 
 ## Measurements own their configuration
 
@@ -212,7 +342,6 @@ A scripted or agent-driven loop:
 
 ```sh
 gauge optimizations diagnose --eval es_1            # where the sessions fail
-gauge optimizations estimate --eval es_1 --target claude-code:claude-opus-4-8
 gauge optimizations start --eval es_1 --target claude-code:claude-opus-4-8 --yes
 gauge optimizations watch cl_1                      # exit 4 = your turn
 
@@ -223,9 +352,10 @@ gauge optimizations trials add cl_1 \
   --body-file start.md --launch --idempotency-key start-env-var --yes
 
 gauge optimizations watch cl_1
-gauge optimizations trial cl_1 R1                   # score, verdict, sessions
+gauge optimizations trial cl_1 R1                   # score, verdict, sessions, errors
 gauge optimizations change cl_1 R1 --body           # the winning text, to paste
 gauge optimizations adopt cl_1 R1 --yes             # mark the winner; baseline stays fixed
+gauge optimizations handoff cl_1                    # instructions to land the winner in your source
 gauge optimizations complete cl_1 --yes
 ```
 
@@ -260,8 +390,14 @@ the answer is you.
 `--idempotency-key` makes `trials add` safe to retry: the same key returns the
 trial the first call created, and never buys a second set of sessions.
 
-Adoption moves Gauge's measured baseline. It does not open a pull request, so
-take the winning text from `change --body` and apply it to your own source.
+There is no round limit or credit budget. Each round waits for you; `complete`
+ends the optimization. Adoption marks the winner and leaves the baseline fixed.
+It does not open a pull request. `handoff` prints the same instructions as the
+web app's "Copy instructions for agent"; follow them in your own source.
+
+A failed or timed-out session drops out of the score. `trial`, `get`, and
+`watch` print a `WARNING` line for each one, with its failure reason; in JSON,
+read `failedSessions` on the trial and `failureReason` on each run.
 
 ## Models, reports, and scripting