@getmagical/cli 0.12.2 → 0.13.0

package/README.md

@@ -80,6 +80,128 @@ The count applies only when creating a regression test. Repeating the command
 for the same automation and agent version pair returns the existing test;
 changing the count does not expand or rerun it.
 
+### Current automation scenarios
+
+Select distinct inputs and available recordings for the scenarios defined by an
+immutable automation version:
+
+```bash
+mgcl pathway-regression coverage --automation-version-id <uuid> --org <org> --max-candidate-runs 1000 --runs-per-scenario 2 --max-recording-checks 100 --json
+```
+
+The supplied version determines the target scenarios. Compatible historical runs
+from any version of the same automation can supply examples. Scenario identity
+uses durable agent pathway keys and required nested calls. Required agents must
+be present; reordered or redundant repeated historical calls do not create new
+scenarios. API agents have one pathway per stable agent ID. Code and Extract
+agents are opaque and pinned to their agent version. Separately queued automation
+runs remain separate candidates.
+
+Discovery uses existing definitions, structured input conditions, and historical
+input witnesses. Feasible scenarios remain visible even without historical
+examples. Missing definitions, unresolved conditions or calls, recursion, and
+expansion limits produce `targetDiagnostics` and incomplete discovery, rather
+than a complete denominator. Expansion is limited to 1,000 scenarios and 10,000
+feasibility evaluations, reported with actual evaluations in `targetDiscovery`.
+This read-only command does not generate definitions,
+backfill signatures, or start runs.
+
+Only record-mode runs with status `COMPLETE` or `EARLY_TERMINATION` qualify,
+including implicit and explicit recordings, with draft/test runs excluded.
+`--max-candidate-runs` accepts 1–10,000 and defaults to **1,000**; the limit applies
+before classification. Compatible signatures identify historical pathways without
+requiring a deviation-free run. Each historical input must also satisfy the
+current scenario's structured conditions; a retained pathway key alone does not
+make an older input applicable after a condition changes. Missing or ambiguous
+evidence appears in
+`unclassifiedRuns`; incompatible historical routes appear in `excludedRuns`.
+Coverage does not reconstruct missing links from raw memories.
+
+`--runs-per-scenario` accepts 1–20 and defaults to **2** distinct inputs. The same
+input is offered only once across the report. JSON object-key order does not
+change input identity; array order and value types do. Store-backed identities
+include immutable starting Store contents. Missing starting data is diagnostic.
+When observations of one input disagree, the newest compatible observation wins,
+with ascending run ID breaking ties; `inputConflicts` records the alternatives.
+
+Repeated runs of one input are alternative recording sources. Selection prefers
+newest verified recordings and falls back to older matching runs. Each recording
+stays paired with its actual source input, work-item ID, and signature evidence.
+`initialStore` preserves immutable starting Store values alongside each example.
+Every JSON example in `selectedRuns` and `inputOnlyRuns` also includes
+`draftRunPayload`, the launch data accepted by `runDraftAutomation`:
+
+```json
+{
+  "input": { "currency": "USD", "paymentMethod": "Cash" },
+  "initialStore": { "request": { "currency": "USD", "paymentMethod": "Cash" } }
+}
+```
+
+The payload contains the source run's original `input` and immutable starting
+`initialStore`, omitting either field when absent. To start a draft run on a clone,
+combine it with the clone's `draftAutomationId`, trigger `agentId`, and a new
+`idempotencyKey` (reuse that key when retrying the same launch). The API creates a
+new work item when work items are enabled; the historical `workItemId` is only
+provenance and must not be reused. With `mgcl automations run-draft`, pass the
+present payload fields as `--input-json` and `--initial-store-json`.
+If migration changes the Store schema, map these values to the clone's new paths
+first: legacy `input` is not automatically converted into Store contents. The
+payload never uses the source work item's mutable final state.
+
+`inputOnlyRuns` preserves matching examples with no verified recording. These
+examples have `recordingId: null` and do not increase recorded sample or coverage
+counts. Each scenario reports `distinctInputCount` and a `gapReason`:
+`no_matching_history`, `recordings_unavailable`, `inputs_assigned_elsewhere`, or
+null when there is no confirmed gap. `inputs_assigned_elsewhere` means compatible
+history exists, but its inputs belong to other scenarios under the newest-observation
+rule; it is a distinct-input shortfall rather than a recording gap. Unchecked recordings and storage errors remain unknown. No compatible
+history yields scenarios with gaps and no fabricated examples.
+
+`--max-recording-checks` accepts 1–10,000 and defaults to **100**. Verification is
+allocated fairly across scenarios, with concurrency 8. A 120-second analysis
+deadline cancels pending database work. No recording archive is downloaded;
+availability does not guarantee future retention or replay fidelity.
+
+Human-readable output includes target pathway keys and labels, required nested
+calls, opaque steps, distinct-input counts, `1/2 selected` recording counts, gaps,
+and diagnostics. Input-only examples include their original inputs and work-item
+IDs. JSON also includes recorded samples' original inputs and signature evidence.
+Errors and warnings go to stderr; `--json` stdout remains valid JSON, including partial results.
+A scenario is covered after its first verified recording, independently of the
+requested sample count.
+
+`coverageStatus` is `complete`, `gaps`, `incomplete`, or `no_scenarios`. Completed
+analysis exits **0**, including genuine gaps and exhausted distinct-input
+shortfalls. When target discovery and recording selection finish and every target
+scenario has a verified recording, coverage is `complete` and exits **0** even if
+some historical runs remain unclassified. These runs remain in `unclassifiedRuns`
+and produce a warning on stderr. Their historical classification diagnostics stay
+in `analysis.incompleteReasons`; they do not invalidate established coverage.
+Incomplete target discovery, unfinished recording selection (including recording
+errors, budget limits, and deadlines), or unclassified history with uncovered
+target scenarios exits **1**, retaining partial results. `analysis.targetDiscoveryComplete`,
+`analysis.historyClassificationComplete`, and `analysis.selectionComplete`
+distinguish these stages. `analysis.discoveryComplete` aggregates target discovery
+and historical classification. Scenario counts are a lower bound only when target
+discovery is incomplete. `no_scenarios` requires complete target discovery with no
+scenarios; an empty history alone does not imply this status. `search.truncated`
+separately identifies older history outside the window and does not itself cause
+failure. Interrupting cancels the GET request without starting background work.
+
+Unsigned agent runs with known baseline disqualifications (human intervention or
+self-heal, including loop recovery) exclude their automation run from examples.
+`excludedRuns` reports `baseline_disqualified` with the reasons in `details`.
+These exclusions do not make classification incomplete, hide current scenarios,
+or refill the historical window with older runs. Missing signatures without a
+known disqualification, and invalid signatures, still make historical
+classification incomplete. They are nonblocking warnings when all target scenarios
+have verified recordings and the remaining analysis has finished.
+
+The command requires only `regressions:read`, `runs:read`, and `captures:read`.
+Deploy backend changes before releasing CLI updates that depend on them. Existing
+`mgcl regression run|show` commands are unchanged.
+
 ## Typed classification
 
 Use `mgcl classify` for focused typed judgments over context you already have.