@withgauge/cli 0.8.0 → 0.9.0

package/README.md

@@ -1,8 +1,8 @@
 # @withgauge/cli
 
 `gauge` — the [Gauge](https://agents.withgauge.com) command-line interface.
-Manage your eval sets, Agent Preference prompts, run presets, and agent runs,
-and watch runs live from your terminal.
+Manage your eval sets, Agent Preference prompts, and agent runs, and watch runs
+live from your terminal.
 
 ```sh
 npm install -g @withgauge/cli
@@ -19,9 +19,8 @@ Requires Node.js ≥ 22.12.
 gauge status                      # the org at a glance
 gauge evals list                  # judged agent-experience measurements
 gauge preference list             # Agent Preference prompts and measurements
-gauge presets list                # run configurations (repo × persona × agents × skills × MCP)
 gauge models list                 # supported model × harness × provider targets
-gauge evals attach <id> --preset <presetId>   # schedule a preset on an eval (prints the bill first)
+gauge evals edit <id> --cadence weekly --samples 3      # when and how much it runs
 gauge evals run <id>              # launch now (ORG-funded, credit-gated)
 gauge runs watch <run-id>         # follow a run as the agent works
 gauge stats rankings -o json      # brand rankings (share of voice) as JSON
@@ -29,17 +28,71 @@ gauge query -d ecosystem -m installs,install_rate -o json   # a flexible report
 ```
 
 Run `gauge --help` (or `gauge <command> --help`) for the full command set:
-evals, preference, presets, models, personas, skills, mcp, cycles, repos, runs,
-experiments, batches, brands, tags, actions, impl, dashboards, members, org,
-provider keys, billing, usage, stats, query, and more.
-
-**Run presets** are the unit of configuration: an eval set or preference
-prompt runs whatever presets are attached to it, every cycle. `gauge presets
-create --name api-main --repo https://github.com/acme/api --agent
-claude-code:claude-opus-4-8 --skill acme-cli@1.2.0` authors one;
-`gauge evals attach` / `gauge preference attach` schedules it. Attaching
-spends money on a schedule, so `attach` and `run` print the per-cycle run
-arithmetic and ask before acting (`--yes` skips the prompt for scripts).
+evals, preference, models, personas, skills, mcp, repos, runs, experiments,
+batches, brands, tags, actions, dashboards, members, org, provider keys,
+billing, usage, stats, query, and more.
+
+## Measurements own their configuration
+
+Every runnable thing carries its own complete run configuration — repository,
+persona, agent roster with model pins, skills, MCP servers, connections — plus
+its own cadence and sample count. Nothing is shared, attached, or detached.
+
+An **eval** is one prompt against one configuration:
+
+```sh
+gauge evals create --name "add a healthcheck" \
+  --text "Add a /health endpoint that returns 200 OK." \
+  --criterion "Returns 200: the endpoint responds 200 with no auth" \
+  --repo https://github.com/acme/api --ref main \
+  --agent claude-code:claude-opus-4-8 --agent codex \
+  --skill acme-cli@1.2.0 \
+  --cadence weekly --samples 3
+```
+
+`gauge evals edit <id>` changes any of it later; omitted flags keep their
+stored value, and each `--clear-*` flag empties one field. To measure the same
+prompt against a second repository or roster, copy the eval in the app and edit
+the copy — one eval is always one configuration.
+
+A **preference prompt** is a reusable question. It owns one set of run
+settings — roster, resources, cadence, samples — and any number of
+**scenarios**, each one repository × persona cell. Every scenario runs with
+the prompt's settings, so one prompt measures the same question across several
+repositories and personas at once:
+
+```sh
+gauge preference create --name "CLI deployment preference" \
+  --kind head-to-head --brand-a acme --brand-b globex \
+  --text "Which CLI should I use to deploy?" \
+  --agent claude-code --cadence daily \
+  --repo https://github.com/acme/api             # first scenario
+
+gauge preference edit <promptId> --agent codex --samples 3 --cadence weekly
+gauge preference scenarios list <promptId>
+gauge preference scenarios add  <promptId> --persona <profileId>
+gauge preference scenarios add  <promptId> --repo https://github.com/acme/web --persona <profileId>
+gauge preference scenarios rm   <promptId> --scenario <scenarioId>
+```
+
+A prompt with zero scenarios is a valid definition — it simply never comes due.
+Sessions per launch = scenarios × agent/model targets × samples. Omit the
+run-settings flags on `create` and the org's default roster applies.
+
+## Cadence is the only off switch
+
+`--cadence none` is "manual": the measurement stops coming due and runs only
+when you launch it. There is no separate pause or enabled flag, in the CLI or
+the app. `--cadence daily|weekly|monthly` puts it back on a rhythm, and
+`preference get` reads a manual prompt back as `cadence: manual`.
+
+`gauge evals run <id>` and `gauge preference run <id>` are the separate
+immediate launches. They use the stored configuration and ask before spending
+(`--yes` skips the prompt for scripts).
+`gauge preference run <id> --scenario <scenarioId>` (repeatable) runs a subset;
+omit it and every scenario runs. Neither launch touches the cadence clock.
+
+## Models, reports, and scripting
 
 Use `gauge models list` before pinning models. It reads the organization's live
 catalog and shows every selectable logical model with its harness, provider,
@@ -47,14 +100,9 @@ and effective capabilities. Open harnesses use `--agent pi:<model>` and
 `--agent opencode:<model>` just like the frontier harnesses.
 
 `gauge query` builds flexible reports over your event warehouse (a GA4
-`run_report` analog): pick a `--dataset`, group by `--dimensions` (including
-`run_preset`), and aggregate `--metrics`, or pass a whole spec with `--json`.
-`gauge query datasets` / `gauge query fields --dataset <name>` list what you
-can ask for.
-
-To launch runs declaratively, `gauge apply -f <spec>.json`. Discover the spec
-format without leaving the terminal: `gauge apply --example` prints a starter
-spec and `gauge apply --schema` prints its JSON Schema.
+`run_report` analog): pick a `--dataset`, group by `--dimensions`, and aggregate
+`--metrics`, or pass a whole spec with `--json`. `gauge query datasets` /
+`gauge query fields --dataset <name>` list what you can ask for.
 
 Every read supports `-o json` for scripting. `GAUGE_API_TOKEN` overrides
 stored credentials (useful in CI). Exit codes: `0` ok · `1` error ·