@getmagical/cli 0.2.5 → 0.2.7

package/README.md

@@ -21,6 +21,145 @@ mgcl automation-runs list --automation-id ...
 mgcl automations run-draft --draft-automation-id ... --agent-id ... --idempotency-key ... --config-json '{"startInTakeOverMode":true,"captureActions":true,"approvalMode":"HUMAN_APPROVAL","consistencyCheckCount":3}'
 ```
 
+## Registered Evals
+
+Registered evals and their persisted runs are global resources, so these
+commands authenticate normally but do not accept `--org`:
+
+```bash
+mgcl evals list --json
+mgcl evals get --slug <eval-slug> --json
+
+# A test run takes one concrete input and is never retried automatically.
+mgcl evals test-run \
+  --slug <eval-slug> \
+  --input-json '{"model":"gpt-5"}' \
+  --json
+
+# A persisted run takes arrays of choices and plans their Cartesian product.
+mgcl evals plan-run \
+  --slug <eval-slug> \
+  --input-file ./eval-inputs.json \
+  --repeat-count 2 \
+  --json
+
+mgcl evals run \
+  --slug <eval-slug> \
+  --input-file ./eval-inputs.json \
+  --repeat-count 2 \
+  --confirm-trials 20 \
+  --wait \
+  --json
+```
+
+Pass exactly one of `--input-file` or `--input-json`. `evals run` always plans
+first. In an interactive terminal it prompts with the exact trial count; JSON
+and other non-interactive invocations must acknowledge that count with
+`--confirm-trials <count>`. The CLI generates an idempotency key when one is
+not supplied and reports it with safe retry guidance if the start request
+fails ambiguously.
+
+Runs start asynchronously unless `--wait` is present. The same bounded wait is
+available independently, with duration units `ms`, `s`, `m`, and `h`:
+
+```bash
+mgcl evals runs list --slug <eval-slug> --status RUNNING --json
+mgcl evals runs get --id <eval-run-id> --json
+mgcl evals runs await --id <eval-run-id> --timeout 10m --json
+mgcl evals runs trials --id <eval-run-id> --limit 100 --cursor <cursor> --json
+mgcl evals runs cancel --id <eval-run-id> --json
+```
+
+Await prints the final structured result before exiting nonzero for `FAILED`,
+`CANCELLED`, or a client timeout. A completed run is operationally successful
+regardless of its scores.
+
+The stable `--json` bodies use the GraphQL field names. Representative shapes
+for the execution lifecycle are:
+
+```json
+{"combinationCount":10,"repeatCount":2,"trialCount":20}
+{"idempotencyKey":"mgcl-eval-...","evalRunId":"...","workflowId":"...","workflowRunId":"..."}
+{"reachedTerminal":true,"pollCount":1,"elapsedMs":42,"evalRun":{"id":"...","status":"COMPLETE","trialStatusCounts":{"queued":0,"running":0,"complete":20,"failed":0,"cancelled":0}}}
+{"trials":[{"id":"...","status":"COMPLETE","input":{},"output":{},"scores":{}}],"nextCursor":"..."}
+```
+
+Definition results include `slug`, `name`, `description`, `tags`, `version`,
+and `inputSchema`. Test results include `status`, `input`, `output`, `scores`,
+and a nullable `{ "type", "message" }` error. Run detail also includes its
+persisted configuration, definition snapshot, progress, timing, initiating
+user, dashboard URL, idempotency key, and Temporal identifiers.
+
+Raw-result exports are asynchronous. `--wait` polls and downloads the completed
+file. Existing files are never overwritten unless `--force` is explicit:
+
+```bash
+mgcl evals runs export \
+  --id <eval-run-id> \
+  --output ./eval-results.json \
+  --wait
+```
+
+Without `--wait`, export JSON reports `exportId`, `evalRunId`, `status`, and
+workflow identifiers. A completed export additionally reports `fileName`,
+`trialCount`, `downloadUrl`, and, after CLI download, the resolved `outputPath`.
+
+These GraphQL operations come from the same shared registry used by the Apollo
+MCP server. MCP callers must provide `idempotencyKey` and
+`expectedTrialCount` directly when starting a run.
+
+## Automation branches
+
+`--branch-id` is a global, per-command authoring target. The CLI never stores a
+current branch. Omit it to target main; branch-aware responses include
+`branchId: null` and the selected main `headVersionId`.
+
+This release adds exactly five commands: branch list, create, get, and merge,
+plus workbench patch readiness.
+
+```bash
+mgcl automations branches list --automation-id <automation-id> --json
+mgcl automations branches create --automation-id <automation-id> --name experiment --json
+mgcl --branch-id <branch-id> automations branches get --automation-id <automation-id> --json
+mgcl --branch-id <branch-id> automations update --id <automation-id> --input-json '{"baseVersionId":"<head-version-id>","name":"Experiment"}' --json
+mgcl --branch-id <branch-id> automations branches merge --automation-id <automation-id> --json
+mgcl --branch-id <branch-id> workbench patch readiness --target-draft-automation-id <automation-id> --operations-json '[{"setAutomationName":{"name":"Experiment readiness preview"}}]' --json
+```
+
+Branch creation defaults to main. Use `--source-branch-id` and optionally
+`--source-version-id` to fork from another branch or an older version in that
+branch's history. Names must use lowercase letters or numbers separated by
+single dashes, such as `release-2`. Spaces, underscores, slashes, backslashes,
+punctuation, uppercase letters, leading or trailing dashes, and repeated dashes
+are rejected before an API request. Branch IDs, not names, identify targets.
+Merge previews the same change set as the web UI. A conflict returns
+`status: "conflict"` and does not change main or delete the branch. A
+conflict-free merge returns `status: "merged"`, the saved-main result, and
+deletes the source branch. Rename, delete, compare, and publish are
+intentionally deferred.
+
+The global branch target is accepted by `automations update`, `automations
+run-draft`, `workbench diagnostics`, `workbench schema-suggestions`, and
+`workbench patch preview`, `readiness`, and `apply`. It is required by
+`automations branches get` and `merge`. Other commands reject `--branch-id`
+before making an API request. Branch-aware responses identify the selected
+target with `branchId` and `headVersionId`. Merge returns `branchId`, its exact
+preview and conflicts, plus the new main head as `applyResult.draftVersionId`
+when successful. Branch inspection excludes workspace authentication records
+and secret values from its hydrated automation response.
+
+For a complete local setup and conflict-free create-to-merge exercise, run the
+commands in [branching-local-test-commands.txt](branching-local-test-commands.txt).
+For a presenter-friendly version that prints every CLI command and pauses for
+approval before merge, run
+[branching-local-demo-commands.txt](branching-local-demo-commands.txt).
+Both scripts derive their service URLs and CLI binary from the current
+worktree, start the tracked portal/main/worker stack, and keep authentication
+in a worktree-local ignored directory. The first run prompts for `mgcl auth
+login`. Set `MGCL_BRANCHING_ORG_ID` to use an organization other than Magical;
+the script headers list the optional namespace, URL, and config-directory
+overrides.
+
 ## Edit a Skill
 
 The Skills working-copy commands currently support root-level skills at