@getmagical/cli 0.2.5 → 0.2.6

package/README.md

@@ -21,6 +21,93 @@ mgcl automation-runs list --automation-id ...
 mgcl automations run-draft --draft-automation-id ... --agent-id ... --idempotency-key ... --config-json '{"startInTakeOverMode":true,"captureActions":true,"approvalMode":"HUMAN_APPROVAL","consistencyCheckCount":3}'
 ```
 
+## Registered Evals
+
+Registered evals and their persisted runs are global resources, so these
+commands authenticate normally but do not accept `--org`:
+
+```bash
+mgcl evals list --json
+mgcl evals get --slug <eval-slug> --json
+
+# A test run takes one concrete input and is never retried automatically.
+mgcl evals test-run \
+  --slug <eval-slug> \
+  --input-json '{"model":"gpt-5"}' \
+  --json
+
+# A persisted run takes arrays of choices and plans their Cartesian product.
+mgcl evals plan-run \
+  --slug <eval-slug> \
+  --input-file ./eval-inputs.json \
+  --repeat-count 2 \
+  --json
+
+mgcl evals run \
+  --slug <eval-slug> \
+  --input-file ./eval-inputs.json \
+  --repeat-count 2 \
+  --confirm-trials 20 \
+  --wait \
+  --json
+```
+
+Pass exactly one of `--input-file` or `--input-json`. `evals run` always plans
+first. In an interactive terminal it prompts with the exact trial count; JSON
+and other non-interactive invocations must acknowledge that count with
+`--confirm-trials <count>`. The CLI generates an idempotency key when one is
+not supplied and reports it with safe retry guidance if the start request
+fails ambiguously.
+
+Runs start asynchronously unless `--wait` is present. The same bounded wait is
+available independently, with duration units `ms`, `s`, `m`, and `h`:
+
+```bash
+mgcl evals runs list --slug <eval-slug> --status RUNNING --json
+mgcl evals runs get --id <eval-run-id> --json
+mgcl evals runs await --id <eval-run-id> --timeout 10m --json
+mgcl evals runs trials --id <eval-run-id> --limit 100 --cursor <cursor> --json
+mgcl evals runs cancel --id <eval-run-id> --json
+```
+
+Await prints the final structured result before exiting nonzero for `FAILED`,
+`CANCELLED`, or a client timeout. A completed run is operationally successful
+regardless of its scores.
+
+The stable `--json` bodies use the GraphQL field names. Representative shapes
+for the execution lifecycle are:
+
+```json
+{"combinationCount":10,"repeatCount":2,"trialCount":20}
+{"idempotencyKey":"mgcl-eval-...","evalRunId":"...","workflowId":"...","workflowRunId":"..."}
+{"reachedTerminal":true,"pollCount":1,"elapsedMs":42,"evalRun":{"id":"...","status":"COMPLETE","trialStatusCounts":{"queued":0,"running":0,"complete":20,"failed":0,"cancelled":0}}}
+{"trials":[{"id":"...","status":"COMPLETE","input":{},"output":{},"scores":{}}],"nextCursor":"..."}
+```
+
+Definition results include `slug`, `name`, `description`, `tags`, `version`,
+and `inputSchema`. Test results include `status`, `input`, `output`, `scores`,
+and a nullable `{ "type", "message" }` error. Run detail also includes its
+persisted configuration, definition snapshot, progress, timing, initiating
+user, dashboard URL, idempotency key, and Temporal identifiers.
+
+Raw-result exports are asynchronous. `--wait` polls and downloads the completed
+file. Existing files are never overwritten unless `--force` is explicit:
+
+```bash
+mgcl evals runs export \
+  --id <eval-run-id> \
+  --output ./eval-results.json \
+  --wait
+```
+
+Without `--wait`, export JSON reports `exportId`, `evalRunId`, `status`, and
+workflow identifiers. A completed export additionally reports `fileName`,
+`trialCount`, `downloadUrl`, and, after CLI download, the resolved `outputPath`.
+
+These GraphQL operations come from the same shared registry used by the Apollo
+MCP server. MCP callers must provide `idempotencyKey` and
+`expectedTrialCount` directly when starting a run.
+
 ## Edit a Skill
 
 The Skills working-copy commands currently support root-level skills at