@getmagical/cli 0.2.4 → 0.2.6

package/README.md

@@ -21,6 +21,129 @@ mgcl automation-runs list --automation-id ...
 mgcl automations run-draft --draft-automation-id ... --agent-id ... --idempotency-key ... --config-json '{"startInTakeOverMode":true,"captureActions":true,"approvalMode":"HUMAN_APPROVAL","consistencyCheckCount":3}'
 ```
 
+## Registered Evals
+
+Registered evals and their persisted runs are global resources, so these
+commands authenticate normally but do not accept `--org`:
+
+```bash
+mgcl evals list --json
+mgcl evals get --slug <eval-slug> --json
+
+# A test run takes one concrete input and is never retried automatically.
+mgcl evals test-run \
+  --slug <eval-slug> \
+  --input-json '{"model":"gpt-5"}' \
+  --json
+
+# A persisted run takes arrays of choices and plans their Cartesian product.
+mgcl evals plan-run \
+  --slug <eval-slug> \
+  --input-file ./eval-inputs.json \
+  --repeat-count 2 \
+  --json
+
+mgcl evals run \
+  --slug <eval-slug> \
+  --input-file ./eval-inputs.json \
+  --repeat-count 2 \
+  --confirm-trials 20 \
+  --wait \
+  --json
+```
+
+Pass exactly one of `--input-file` or `--input-json`. `evals run` always plans
+first. In an interactive terminal it prompts with the exact trial count; JSON
+and other non-interactive invocations must acknowledge that count with
+`--confirm-trials <count>`. The CLI generates an idempotency key when one is
+not supplied and reports it with safe retry guidance if the start request
+fails ambiguously.
+
+Runs start asynchronously unless `--wait` is present. The same bounded wait is
+available independently, with duration units `ms`, `s`, `m`, and `h`:
+
+```bash
+mgcl evals runs list --slug <eval-slug> --status RUNNING --json
+mgcl evals runs get --id <eval-run-id> --json
+mgcl evals runs await --id <eval-run-id> --timeout 10m --json
+mgcl evals runs trials --id <eval-run-id> --limit 100 --cursor <cursor> --json
+mgcl evals runs cancel --id <eval-run-id> --json
+```
+
+Await prints the final structured result before exiting nonzero for `FAILED`,
+`CANCELLED`, or a client timeout. A completed run is operationally successful
+regardless of its scores.
+
+The stable `--json` bodies use the GraphQL field names. Representative shapes
+for the execution lifecycle are:
+
+```json
+{"combinationCount":10,"repeatCount":2,"trialCount":20}
+{"idempotencyKey":"mgcl-eval-...","evalRunId":"...","workflowId":"...","workflowRunId":"..."}
+{"reachedTerminal":true,"pollCount":1,"elapsedMs":42,"evalRun":{"id":"...","status":"COMPLETE","trialStatusCounts":{"queued":0,"running":0,"complete":20,"failed":0,"cancelled":0}}}
+{"trials":[{"id":"...","status":"COMPLETE","input":{},"output":{},"scores":{}}],"nextCursor":"..."}
+```
+
+Definition results include `slug`, `name`, `description`, `tags`, `version`,
+and `inputSchema`. Test results include `status`, `input`, `output`, `scores`,
+and a nullable `{ "type", "message" }` error. Run detail also includes its
+persisted configuration, definition snapshot, progress, timing, initiating
+user, dashboard URL, idempotency key, and Temporal identifiers.
+
+Raw-result exports are asynchronous. `--wait` polls and downloads the completed
+file. Existing files are never overwritten unless `--force` is explicit:
+
+```bash
+mgcl evals runs export \
+  --id <eval-run-id> \
+  --output ./eval-results.json \
+  --wait
+```
+
+Without `--wait`, export JSON reports `exportId`, `evalRunId`, `status`, and
+workflow identifiers. A completed export additionally reports `fileName`,
+`trialCount`, `downloadUrl`, and, after CLI download, the resolved `outputPath`.
+
+These GraphQL operations come from the same shared registry used by the Apollo
+MCP server. MCP callers must provide `idempotencyKey` and
+`expectedTrialCount` directly when starting a run.
+
+## Edit a Skill
+
+The Skills working-copy commands currently support root-level skills at
+`skills/<skill-name>`:
+
+```bash
+mgcl skills pull review
+# Edit ./skills/review in your editor.
+mgcl skills diff review
+mgcl skills push review --message "Improve review guidance"
+```
+
+`pull` downloads the complete skill directory from the repository's default
+branch and remembers the pulled contents. Tags, commit SHAs, and non-default
+branches remain available through the low-level read/history commands, but are
+not supported by the writable working-copy workflow.
+It refuses to overwrite existing files or local edits unless you pass `--force`.
+`diff` compares your files with the pulled contents. `push` sends all
+additions, edits, and deletions as one commit against the current repository
+head. After a successful push, the working copy and its remembered baseline
+are replaced with the committed bundle read back at the new commit, so they
+include any concurrent change to another file in the same skill. If that
+refresh fails, the push still succeeded; re-run
+`mgcl skills pull <skill> --force` to sync.
+
+These commands do not protect concurrent edits to the same skill. If the skill
+changed remotely and your local patch no longer applies, `push` fails and asks
+you to pull the latest version and reapply your changes. Binary file changes
+cannot be pushed.
+
+These commands use temporary Git repositories only; they do not initialize or
+modify Git state in the directory where you run `mgcl`.
+
+For low-level automation, `mgcl skills apply-patch` applies one raw Git patch
+as a single commit.
+
 ## Local Development
 
 Run the CLI from source without building: