@getmagical/cli 0.4.0 → 0.7.0

package/README.md

@@ -12,15 +12,376 @@ Magical CLI for the shared MCP operation surface.
 ```bash
 mgcl --version
 mgcl update
+mgcl settings env --env uk
 mgcl auth login
-mgcl settings env base-url http://main.localhost:1355
 mgcl org list
 mgcl automations list --limit 10
 mgcl automations update --id ... --input-json ...
 mgcl automation-runs list --automation-id ...
 mgcl automations run-draft --draft-automation-id ... --agent-id ... --idempotency-key ... --config-json '{"startInTakeOverMode":true,"captureActions":true,"approvalMode":"HUMAN_APPROVAL","consistencyCheckCount":3}'
+mgcl automations publish --draft-automation-id ...   # non-production only: mgcl refuses against the production API, and the API rejects it too
 ```
 
+## Deployment targets
+
+Use the global `--env us|uk|stg` option to select a complete Magical API and
+WorkOS authentication target for one command:
+
+```bash
+mgcl --env us org list
+mgcl --env uk auth login
+mgcl --env uk automations list --json
+mgcl --env stg org list
+```
+
+The built-in presets contain only public deployment configuration:
+
+| Preset | Platform | Magical API | WorkOS client |
+| --- | --- | --- | --- |
+| `us` | `https://platform.getmagical.io` | `https://api-agt.getmagical.io` | `client_01JJZ6XJ1RF248P20WF63M4VAM` |
+| `uk` | `https://platform.uk.getmagical.io` | `https://api-agt.uk.getmagical.io` | `client_01M0J897D67RXRB4VBNTXBF9ST` |
+| `stg` | `https://platform-staging.getmagical.net` | `https://api-agt-staging.getmagical.net` | `client_01JJZ6X26BGFBT8AC303XPQ9EQ` |
+
+All three use the public WorkOS API at `https://api.workos.com`. A platform
+domain in a built-in preset resolves to its corresponding Magical API origin
+before requests are sent.
+
+For an explicit target, combine `--domain` with `--workosEnv`. `--workosEnv`
+accepts the `us`, `uk`, or `stg` WorkOS configuration alias, or a public WorkOS
+client ID beginning with `client_`. It never accepts or requires a WorkOS API
+key.
+
+```bash
+mgcl \
+  --domain https://platform.uk.getmagical.io \
+  --workosEnv uk \
+  auth login
+
+mgcl \
+  --domain https://api.example.internal \
+  --workosEnv client_01ABCDEF123456789 \
+  org list
+```
+
+An unknown custom domain needs a deliberately selected WorkOS environment,
+either from saved settings or the same command. A domain-only override keeps a
+saved explicit `--env` or `--workosEnv` selection, but it never inherits the
+implicit built-in US default or infers staging from the presence of a custom
+URL.
+
+### Save a default target
+
+Persist the same selectors with `mgcl settings env`, inspect the saved and
+effective target with no options, or reset to the built-in US default:
+
+```bash
+mgcl settings env --env uk
+mgcl settings env
+mgcl auth login
+
+mgcl settings env \
+  --domain https://platform.uk.getmagical.io \
+  --workosEnv uk
+
+mgcl settings env --reset
+```
+
+A global selector applies only to its invocation and never rewrites saved
+settings. An explicit global `--env` starts from that complete preset rather
+than inheriting saved overrides; global `--domain` and `--workosEnv` then
+override the selected target for that command.
+
+Saved interactive sessions are bound to the resolved API and WorkOS target.
+When the selected target does not match that saved session, mgcl requires
+authentication for the selected target instead of silently reusing it.
+Changing saved settings clears incompatible saved authentication; saving the
+same effective target preserves it. Explicit and environment-provided API keys
+remain invocation credentials and are validated by the selected deployment.
+
+The legacy `mgcl settings env base-url <url>` command and saved `env.baseUrl`
+values remain supported for recognized preset URLs. A legacy unknown URL with
+no deliberately saved WorkOS selection cannot identify its authentication
+environment safely and must be migrated explicitly.
+For local or preview services that authenticate against staging WorkOS, use:
+
+```bash
+mgcl settings env \
+  --domain http://main.localhost:1355 \
+  --workosEnv stg
+mgcl auth login
+```
+
+This native configuration replaces the temporary workaround of patching an
+installed mgcl package to select the UK WorkOS client.
+
+## Run evidence and private files
+
+Discover immutable S3 references before downloading a specific file:
+
+```bash
+mgcl automation-runs files --id <automation-run-id> --org <org> --json
+mgcl files get --key <s3-key> --download ./local-file --org <org> --json
+```
+
+`automation-runs files` scans run input, agent input, and paged tool-call and
+tool-result memories. Its JSON output identifies each file's agent run,
+iteration, field path, role, source, tool call, and tool name, plus whether the
+scan was truncated. `files get` downloads into the current directory using the
+original file name when `--download` is omitted. It refuses to replace an
+existing destination unless `--force` is present.
+
+Use the dedicated commands for final agent output and database-backed tool
+result media:
+
+```bash
+mgcl agent-runs output --id <agent-run-id> --org <org> --json
+mgcl automation-runs iteration-media \
+  --iteration-id <iteration-id> \
+  --media-id <media-id> \
+  --download ./tool-result-media \
+  --org <org> \
+  --json
+```
+
+Private download URLs are transport details. The CLI writes file bytes only to
+the requested local path and omits signed or authenticated URLs from both human
+and JSON output. These read paths require `runs:read` when an API key is used.
+
+Knowledge Base and Memory discovery stays metadata-only. Retrieve a selected
+immutable version, or upload an ordinary replay input, through mgcl's private
+file transport:
+
+```bash
+mgcl knowledge read --key <repository-file-key> --download ./knowledge-file
+mgcl memories read --key <repository-file-key> --download ./memory-file
+mgcl files upload --path ./run-input.pdf
+```
+
+`files get`, `automation-runs iteration-media`, and these three transfer
+commands are mgcl-only. They are omitted from the generated raw MCP tool list
+so signed download and upload URLs never become normal MCP output. The GraphQL
+operations remain available to mgcl's private transport.
+
+For verification, download evidence into a private session-specific temporary
+directory. Do not echo or paste raw file contents or download URLs into the
+transcript, and remove the files during session cleanup. If a file is missing,
+record that evidence as unverifiable instead of inferring its contents.
+
+Inspect a retained network recording by its authoritative automation run ID:
+
+```bash
+mgcl network-recordings get --automation-run-id <run-id> --org <org> --json
+mgcl network-recordings get --automation-run-id <run-id> --output ./recording.mitm --org <org> --json
+mgcl network-recordings exchanges --automation-run-id <run-id> --limit 100 --org <org> --json
+mgcl network-recordings response --automation-run-id <run-id> --exchange <exchange-ref> --output ./response.pdf --org <org> --json
+```
+
+`get` is metadata-only unless `--output` intentionally requests the complete
+archive. `exchanges` and `response` privately download a fresh archive for that
+invocation and remove it after local parsing. This keeps the organization, run,
+and recording association authoritative without accepting archive imports.
+Downloads are capped at 256 MiB, require the received byte count to match the
+availability metadata, use private file permissions, and refuse an existing
+destination. Metadata lookup has a 30-second deadline. Complete archive
+transfer, including response-body streaming, has a separate 120-second
+deadline.
+
+Local inspection uses mitmproxy's supported archive reader and requires Python
+3.12 or newer with exactly `mitmproxy==12.2.3`. mgcl does not install this
+optional runtime. Create a dedicated environment and point mgcl at its Python
+executable:
+
+```bash
+python3 -m venv ~/.local/share/mgcl/mitmproxy-12.2.3
+~/.local/share/mgcl/mitmproxy-12.2.3/bin/python -m pip install 'mitmproxy==12.2.3'
+export MGCL_MITMPROXY_PYTHON=~/.local/share/mgcl/mitmproxy-12.2.3/bin/python
+```
+
+Exchange output includes a recording-specific stable reference, flow ordinal,
+method, request origin, response status, normalized content media type,
+and body availability. Request paths, query strings, user info, headers, and
+bodies are omitted. The scan reports separate result and 20,000-flow limits,
+counts, and whether it was complete. Response extraction verifies the exchange
+reference against the freshly downloaded archive and reports the decoded entity
+body's size and SHA-256 provenance without printing its contents.
+
+Inspection has a 60-second processing deadline and rejects an archive over 256
+MiB or an individual serialized flow over 32 MiB. Extraction additionally
+rejects encoded bodies over 32 MiB, decoded bodies over 64 MiB, or decompression
+over 100 times the encoded size. Version 1 extraction supports an absent or
+`identity` content encoding, `gzip`, and `deflate`. Brotli (`br`), Zstandard
+(`zstd`), unknown, and stacked encodings return `UNSUPPORTED_ENCODING` without
+creating an output file. Truncated, trailing, or concatenated gzip and deflate
+streams are invalid. A missing response or body, an unsupported encoding, a
+malformed archive, or any exceeded bound means the requested evidence cannot be
+verified; it does not establish that the recorded automation failed.
+
+## Cloud authentication
+
+Use a WorkOS user API key for unattended workloads. Create keys from a trusted
+laptop with a saved interactive login; do not run the setup wizard in the
+cloud workload itself:
+
+```bash
+mgcl auth login
+mgcl auth setup-cloud --output ./magical-cloud.env
+```
+
+The wizard refreshes the interactive session, discovers active organization
+memberships, and lets you select one or more organizations, a purpose, an
+`inspect`, `build`, or `operate` preset, all available permissions, or exact
+scopes, an optional expiry, and one default organization. Interactive sessions
+use arrow-key and checkbox prompts. The wizard displays the owner,
+organizations, grants, expiry, and destination for confirmation before creating
+anything. Keys do not expire by default; choose an expiry when the workload has
+a known lifetime.
+
+For non-interactive setup, pass `--permissions all` to expand to every
+permission supported by that CLI version. `all` is a CLI shortcut and is never
+sent to WorkOS as a permission.
+
+The output is a sourceable environment file. Copy its values into the cloud
+provider's secret store rather than committing the file. Each selected
+organization gets one key and an organization-specific variable:
+
+```bash
+MAGICAL_API_KEY_ORG_01JTH6STB70QTCXJWVJE22G9J3=...
+```
+
+The suffix is the uppercase ULID from the canonical organization ID, without
+the `org_` prefix. The selected default organization's same key is also written
+as `MAGICAL_API_KEY`; this alias does not create another key.
+
+For a one-command override, pass the global `--api-key <key>` option before or
+after the command path:
+
+```bash
+mgcl automations list --api-key "$ONE_TIME_MAGICAL_API_KEY"
+```
+
+The key is ephemeral: it is not written to CLI state, preferences, the
+keychain, or the environment (`org use` may still save the non-secret
+organization choice). It is the only credential candidate for that invocation,
+so the CLI ignores all `MAGICAL_API_KEY` variables and saved login credentials
+rather than falling back after a failure. With `--org`, the key must belong to
+that exact organization. Literal command-line values can be visible in shell
+history and process listings, so prefer expansion from a secret manager or
+environment variable and continue using the canonical environment variables
+for unattended workloads.
+
+### Organization selection
+
+When any `MAGICAL_API_KEY` variable is configured, workspace commands use API
+key mode and never fall back to a saved login or direct access token. Selection
+is deterministic:
+
+1. An explicit `--org <organization>`.
+2. The organization saved for the current API origin by `mgcl org use`.
+3. The organization validated from `MAGICAL_API_KEY`.
+4. The only valid `MAGICAL_API_KEY_ORG_*` key, when exactly one is available.
+
+For a selected organization, its exact organization-specific variable wins.
+If that variable exists but is invalid, expired, or insufficient, the command
+fails instead of trying `MAGICAL_API_KEY` or another identity. When it is
+absent, `MAGICAL_API_KEY` is eligible only if its metadata identifies that same
+organization. The CLI validates identity and necessary permissions before the
+business request and never retries that request under another key. Organization
+discovery also rejects configured keys owned by different users.
+
+Use live, sanitized metadata to inspect or choose an organization:
+
+```bash
+mgcl org list
+mgcl org use <org-id-or-exact-name>
+mgcl auth status
+```
+
+The saved choice contains no credential and is isolated by API origin.
+`auth status` reports the active authentication source plus key and organization
+metadata, never the raw key. `mgcl auth logout` clears saved interactive auth
+and a direct access token, but it cannot unset environment variables or revoke
+their keys; configured environment keys remain active.
+
+### Manage, rotate, and recover keys
+
+Key management always requires and refreshes the saved interactive session. It
+deliberately ignores environment API keys and direct access tokens, and rejects
+`--api-key` so a management action cannot silently run under another identity:
+
+```bash
+mgcl auth keys list --org <org>
+mgcl auth keys revoke <key-id> --org <org>
+```
+
+Listing returns sanitized metadata, including grants, expiry, last use, and an
+obfuscated value. WorkOS returns a raw value only once, during creation. If the
+export is lost, it cannot be recovered: revoke that key and create a
+replacement. To rotate safely, create and install the replacement, verify the
+workload with it, and then revoke the old key. Start a rotation with a new
+output path (and therefore a new default checkpoint) so the completed original
+setup remains an audit and recovery record:
+
+```bash
+mgcl auth setup-cloud --output ./magical-cloud-rotation-2026-09.env
+```
+
+Rerunning a completed setup checkpoint is intentionally a no-op, and `--force`
+cannot restart or change that saved plan. Use a new `--output`, or both a new
+`--output` and `--checkpoint`, for every rotation.
+
+Setup writes every returned secret immediately and durably, and never prints
+one to the terminal. Its output and sidecar checkpoint are owner-only files
+(`0600`); the checkpoint contains recovery metadata but no raw key. Keep both
+private, and rerun the same setup command after an interruption so it can use
+the checkpoint and idempotency records to resume or reconcile an uncertain
+attempt without blindly creating duplicates. Existing output is not replaced
+without explicit overwrite confirmation.
+
+Keys belong to the API environment where they were created. Configure a custom
+origin before login and setup, and use separate output and secret records for
+local, staging, preview, and production environments:
+
+```bash
+mgcl settings env base-url https://api.example.internal
+mgcl auth login
+mgcl auth setup-cloud --output ./example-cloud.env
+```
+
+Changing or resetting the base URL clears saved interactive auth. API requests
+also reject redirects instead of forwarding a bearer credential to another
+origin.
+
+Cloud keys are least-privilege automation credentials. Their effective access
+is the intersection of the key's scopes and the owner's current WorkOS role;
+an active membership is required on every request. Unsupported surfaces remain
+denied, including eval management, publishing and deletion, schedules, admin
+operations, raw credential values, browser sessions, interactive sessions, and
+key minting.
+
+### Operator rollout and rollback
+
+Deploy the backend before releasing the CLI flow. In WorkOS, create these
+permissions and grant the appropriate subset to member roles:
+
+```text
+automations:read automations:write runs:read runs:execute runs:control
+store:read store:write skills:read skills:write credentials:read
+captures:read captures:write regressions:read regressions:run
+knowledge:read memory:read files:write
+```
+
+The Knowledge Base, Memory, and replay-upload commands remain pending until the
+backend API is deployed and these three permission slugs are configured on the
+intended WorkOS roles.
+
+Roles that manage their own keys also need WorkOS's
+`widgets:user-api-keys:manage-self` permission. Then deploy and verify the
+backend before releasing the cloud-auth CLI. The backend defaults
+`WORKOS_USER_API_KEYS_ENABLED` to `true`; setting it to `false` is the emergency
+rollback. While disabled, the backend rejects API-key authentication and new
+cloud setup, while interactive login and interactive key listing/revoking
+remain available so existing WorkOS records can be managed manually.
+
 ## Registered Evals
 
 Registered evals and their persisted runs are global resources, so these
@@ -114,18 +475,26 @@ MCP server. MCP callers must provide `idempotencyKey` and
 current branch. Omit it to target main; branch-aware responses include
 `branchId: null` and the selected main `headVersionId`.
 
-This release adds exactly five commands: branch list, create, get, and merge,
-plus workbench patch readiness.
+Branch commands: list, create, get, merge-preview, discard-changes, and
+merge, plus
+workbench patch readiness.
 
 ```bash
 mgcl automations branches list --automation-id <automation-id> --json
 mgcl automations branches create --automation-id <automation-id> --name experiment --json
 mgcl --branch-id <branch-id> automations branches get --automation-id <automation-id> --json
 mgcl --branch-id <branch-id> automations update --id <automation-id> --input-json '{"baseVersionId":"<head-version-id>","name":"Experiment"}' --json
+mgcl --branch-id <branch-id> automations branches merge-preview --automation-id <automation-id> --json
+mgcl --branch-id <branch-id> automations branches discard-changes --automation-id <automation-id> --change-ids agent:update:<agent-id>:instructions --source-version-id <source-version-id> --target-version-id <target-version-id> --json
 mgcl --branch-id <branch-id> automations branches merge --automation-id <automation-id> --json
 mgcl --branch-id <branch-id> workbench patch readiness --target-draft-automation-id <automation-id> --operations-json '[{"setAutomationName":{"name":"Experiment readiness preview"}}]' --json
 ```
 
+Merge previews report agent edits per field, and settings per config key:
+`agent:update:<agent-id>:config.model`, `config.tools`, `config.plugins.memory`,
+or `config.devOnly` for the developer-only keys, with the key in `configKey`.
+Automation settings follow the same shape (`automation:config.systemPrompt`).
+
 Branch creation defaults to main. Use `--source-branch-id` and optionally
 `--source-version-id` to fork from another branch or an older version in that
 branch's history. Names must use lowercase letters or numbers separated by
@@ -135,7 +504,24 @@ leading or trailing dashes, repeated dashes, and over-length names are
 rejected before an API request. Branch IDs, not names, identify targets, and
 every automation or branch ID must be a UUID; malformed IDs are rejected
 before an API request.
-Merge previews the same change set as the web UI. A conflict returns
+`merge-preview` explains the merge without writing anything. Each change carries a
+stable `id`, a `kind`, and for agent edits the `agentId` and `field`
+(`name`, `description`, `instructions`, `inputSchema`, `outputSchema`,
+`config`, `agentLibraryEntity`, `type`), so a field main also changed
+conflicts on its own while the agent's other fields stay mergeable.
+Connections are keyed by `fromAgentId`/`toAgentId`, agent positions are one
+grouped `layout:agents` change that never conflicts, and a connection to an
+agent main deleted is a conflict. `conflictDescription` says why a change
+conflicts; `discardable` marks changes that can be reverted.
+
+`discard-changes` is the only conflict resolution: it reverts the named changes on
+the branch as a new branch version so main keeps its values, and returns the
+new head plus the refreshed preview. Pass `--source-version-id` and
+`--target-version-id` from the preview you acted on; the command is rejected
+when either head moved since. Discarding an added or removed agent also
+reverts its added or removed connections.
+
+`merge` previews the same change set as the web UI. A conflict returns
 `status: "conflict"`, does not change main or delete the branch, and exits
 nonzero so scripts gating on the exit code never mistake it for an applied
 merge. A conflict-free merge returns `status: "merged"`, the saved-main
@@ -145,7 +531,8 @@ are intentionally deferred.
 The global branch target is accepted by `automations update`, `automations
 run-draft`, `automations diff-from-published`, `workbench diagnostics`,
 `workbench schema-suggestions`, and `workbench patch preview`, `readiness`,
-and `apply`. It is required by `automations branches get` and `merge`. Other
+and `apply`. It is required by `automations branches get`, `merge-preview`,
+`discard-changes`, and `merge`. Other
 commands reject `--branch-id` before making an API request, and
 `mgcl --help` lists the branch-aware commands. Branch-aware responses identify the selected
 target with `branchId` and `headVersionId`. Merge returns `branchId`, its exact
@@ -209,23 +596,20 @@ Run the CLI from source without building:
 npm run cli -- --help
 ```
 
-Production installs use the bundled WorkOS client ID by default. To persist a
-local or staging API target, set:
+Use an isolated configuration directory when testing a local or preview target
+without changing the normal CLI profile:
 
 ```bash
-mgcl settings env base-url http://main.localhost:1355
+export MAGICAL_CLI_CONFIG_DIR="$(mktemp -d)"
+export MAGICAL_CLI_REFRESH_TOKEN_STORE=file
+mgcl settings env --domain http://main.localhost:1355 --workosEnv stg
 mgcl auth login
 ```
 
-When a persisted base URL override is present, `mgcl auth login` uses the
-bundled staging/dev WorkOS client ID automatically. Changing or resetting the
-override clears saved auth so you can log into the correct environment again.
-
-To remove the override and return to defaults:
-
-```bash
-mgcl settings env base-url --reset
-```
+Complete the browser flow only when that isolated profile is ready for use,
+then run `mgcl org list` and one read-only command against the same target.
+Move the temporary directory to Trash after the test credentials are no longer
+needed: `trash "$MAGICAL_CLI_CONFIG_DIR"`.
 
 ## Install Script
 
@@ -249,6 +633,13 @@ Once installed, users can update from the same channel with:
 mgcl --update
 ```
 
+The service checks the npm `latest` version dynamically for mgcl requests, so
+an older version observed during a rollout is not a permanent minimum. Release
+new CLI behavior by merging it with its package-version bump, deploying any
+required server support first, then creating the matching `cli-v<version>`
+GitHub release and verifying that npm `latest` and each target deployment accept
+the published version.
+
 To remove the managed install:
 
 ```bash