@eigenpal/cli 0.14.3 → 0.15.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (19) hide show
  1. package/CHANGELOG.md +11 -0
  2. package/README.md +11 -9
  3. package/dist/cli.js +1110 -249
  4. package/dist/docs-bundle.json +1356 -0
  5. package/dist/skill/SKILL.md +33 -0
  6. package/dist/skill/reference/cli/agents.md +111 -106
  7. package/dist/skill/reference/cli/docs.md +84 -0
  8. package/dist/skill/reference/cli/models.md +1 -1
  9. package/dist/skill/reference/cli/rerun.md +1 -1
  10. package/dist/skill/reference/cli/run.md +1 -1
  11. package/dist/skill/reference/cli/runs.md +67 -67
  12. package/dist/skill/reference/cli/surface.md +4 -0
  13. package/dist/skill/reference/cli/workflow.md +114 -107
  14. package/dist/skill/reference/dataset-format.md +151 -0
  15. package/dist/skill/reference/debugging.md +9 -0
  16. package/dist/skill/reference/evaluators.md +361 -4
  17. package/dist/skill/reference/step-types.md +4519 -15
  18. package/dist/skill/reference/workflow-yaml.md +1458 -23
  19. package/package.json +7 -4
package/dist/skill/reference/cli/workflow.md CHANGED
@@ -118,15 +118,15 @@ workflow
118
118
 
119
119
  ### Core
120
120
 
121
- | Command | Description |
122
- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
123
- | `eigenpal workflow list\|ls [options]` | List workflows the caller can read. |
124
- | `eigenpal workflow pull [options] <workflow-id>` | Download the YAML definition of the workflow at its current version. |
125
- | `eigenpal workflow schema [options] <workflow-id>` | Show the inferred output schema for a workflow (what it returns). |
126
- | `eigenpal workflow push [options]` | Create or update a workflow from a YAML file. |
127
- | `eigenpal workflow move [options] <workflow-id>` | Move a workflow to a folder path, creating folders as needed |
128
- | `eigenpal workflow validate [options] [path]` | Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/). With [path] pointing at a YAML file: validates that workflow.yaml. For per-noun targeting use `evaluators validate` or `dataset validate`. |
129
- | `eigenpal workflow clear-local [options] [examples...]` | Delete local execution artifacts under ./dataset/examples/. Keeps the latest run per example by default. |
121
+ | Command | Description |
122
+ | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
123
+ | `eigenpal workflow list\|ls [options]` | List workflows the caller can read. |
124
+ | `eigenpal workflow pull [options] <workflow-id>` | Download the YAML definition of the workflow at its current version. |
125
+ | `eigenpal workflow schema [options] <workflow-id>` | Show the inferred output schema for a workflow (what it returns). |
126
+ | `eigenpal workflow push [options]` | Create or update a workflow from a YAML file. |
127
+ | `eigenpal workflow move [options] <workflow-id>` | Move a workflow to a folder path, creating folders as needed |
128
+ | `eigenpal workflow validate [options] [path]` | Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/) in the project root. When the root has no workflow.yaml, discovers nested projects under eigenpal/workflows/<slug>/ or workflows/<slug>/ and validates each. With [path] pointing at a YAML file: validates that workflow.yaml only. For per-noun targeting use `evaluators validate` or `dataset validate`. |
129
+ | `eigenpal workflow clear-local [options] [examples...]` | Delete local execution artifacts under ./dataset/examples/. Keeps the latest run per example by default. |
130
130
 
131
131
  ### Evaluators
132
132
 
@@ -170,7 +170,7 @@ workflow
170
170
  | `eigenpal workflow experiment\|exp status [options] <workflow-id> <batchId>` | Aggregate progress for a batch by `batchId`. |
171
171
  | `eigenpal workflow experiment\|exp cancel [options] <workflow-id> <batchId>` | Cancel every execution in a batch. Idempotent. |
172
172
  | `eigenpal workflow experiment\|exp results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON. |
173
- | `eigenpal workflow experiment\|exp compare\|diff [options] <batchIdA> <batchIdB>` | Diff eval scores between two experiment batches. |
173
+ | `eigenpal workflow experiment\|exp compare\|diff [options] <batchIdA> <batchIdB>` | Compare evaluator scores or actual outputs between two experiment batches. |
174
174
  | `eigenpal workflow experiment\|exp watch [options] <workflow-id> <batchId>` | Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`. |
175
175
 
176
176
  ### Versions
@@ -210,13 +210,13 @@ List workflows the caller can read.
210
210
 
211
211
  ### Options
212
212
 
213
- | Flag | Required | Default | Description |
214
- | ------------------ | -------- | ------- | -------------------------------------- |
215
- | `--search <q>` | no | | Filter by name |
216
- | `--limit <n>` | no | `50` | Page size |
217
- | `--offset <n>` | no | `0` | Page offset |
218
- | `--base-url <url>` | no | | Server base URL |
219
- | `--json` | no | | Output the raw server response as JSON |
213
+ | Flag | Required | Default | Description |
214
+ | ------------------ | -------- | ------- | ------------------------------------ |
215
+ | `--search <q>` | no | | Filter by name |
216
+ | `--limit <n>` | no | `50` | Page size |
217
+ | `--offset <n>` | no | `0` | Page offset |
218
+ | `--base-url <url>` | no | | Server base URL |
219
+ | `--json` | no | | Emit machine-readable JSON on stdout |
220
220
 
221
221
  ### `eigenpal workflow pull [options] <workflow-id>`
222
222
 
@@ -267,7 +267,7 @@ Create or update a workflow from a YAML file.
267
267
  | `--set-version <semver>` | no | | Push at this exact semver (e.g. 1.4.0). Mutually exclusive with `--bump` and with a top-level `version:` in the YAML. (Named `--set-version` to avoid the global `-v, --version` flag.) |
268
268
  | `--allow-external-templates` | no | | Allow local template: paths whose real path is outside the workflow project directory (the folder that contains the YAML file). Off by default; ../ and symlink escapes are rejected. |
269
269
  | `--base-url <url>` | no | | Server base URL |
270
- | `--json` | no | | Output the raw server response as JSON |
270
+ | `--json` | no | | Emit machine-readable JSON on stdout |
271
271
 
272
272
  ### `eigenpal workflow move [options] <workflow-id>`
273
273
 
@@ -281,15 +281,15 @@ Move a workflow to a folder path, creating folders as needed
281
281
 
282
282
  ### Options
283
283
 
284
- | Flag | Required | Default | Description |
285
- | ------------------ | -------- | ------- | -------------------------------------- |
286
- | `--folder <path>` | yes | | Target folder path (`/` for root) |
287
- | `--base-url <url>` | no | | Server base URL |
288
- | `--json` | no | | Output the raw server response as JSON |
284
+ | Flag | Required | Default | Description |
285
+ | ------------------ | -------- | ------- | ------------------------------------ |
286
+ | `--folder <path>` | yes | | Target folder path (`/` for root) |
287
+ | `--base-url <url>` | no | | Server base URL |
288
+ | `--json` | no | | Emit machine-readable JSON on stdout |
289
289
 
290
290
  ### `eigenpal workflow validate [options] [path]`
291
291
 
292
- Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/). With [path] pointing at a YAML file: validates that workflow.yaml. For per-noun targeting use `evaluators validate` or `dataset validate`.
292
+ Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/) in the project root. When the root has no workflow.yaml, discovers nested projects under eigenpal/workflows/<slug>/ or workflows/<slug>/ and validates each. With [path] pointing at a YAML file: validates that workflow.yaml only. For per-noun targeting use `evaluators validate` or `dataset validate`.
293
293
 
294
294
  ### Arguments
295
295
 
@@ -352,11 +352,11 @@ Overwrite the workflow's evaluator config from a YAML file.
352
352
 
353
353
  ### Options
354
354
 
355
- | Flag | Required | Default | Description |
356
- | ------------------ | -------- | ------- | -------------------------------------- |
357
- | `--file <yaml>` | yes | | Path to evaluators YAML file |
358
- | `--base-url <url>` | no | | Server base URL |
359
- | `--json` | no | | Output the raw server response as JSON |
355
+ | Flag | Required | Default | Description |
356
+ | ------------------ | -------- | ------- | ------------------------------------ |
357
+ | `--file <yaml>` | yes | | Path to evaluators YAML file |
358
+ | `--base-url <url>` | no | | Server base URL |
359
+ | `--json` | no | | Emit machine-readable JSON on stdout |
360
360
 
361
361
  ### `eigenpal workflow evaluators validate [options] [path]`
362
362
 
@@ -380,12 +380,12 @@ List eval examples for the workflow.
380
380
 
381
381
  ### Options
382
382
 
383
- | Flag | Required | Default | Description |
384
- | ------------------ | -------- | ------- | -------------------------------------- |
385
- | `--limit <n>` | no | `100` | Page size |
386
- | `--offset <n>` | no | `0` | Page offset |
387
- | `--base-url <url>` | no | | Server base URL |
388
- | `--json` | no | | Output the raw server response as JSON |
383
+ | Flag | Required | Default | Description |
384
+ | ------------------ | -------- | ------- | ------------------------------------ |
385
+ | `--limit <n>` | no | `100` | Page size |
386
+ | `--offset <n>` | no | `0` | Page offset |
387
+ | `--base-url <url>` | no | | Server base URL |
388
+ | `--json` | no | | Emit machine-readable JSON on stdout |
389
389
 
390
390
  ### `eigenpal workflow dataset pull [options] <workflow-id>`
391
391
 
@@ -423,7 +423,7 @@ Replace or extend the workflow's dataset from a ZIP or folder.
423
423
  | `--mode <append\|replace>` | no | `"append"` | Import mode (default: append) |
424
424
  | `--yes` | no | `false` | Skip the destructive confirmation prompt for --mode replace (use in CI) |
425
425
  | `--base-url <url>` | no | | Server base URL |
426
- | `--json` | no | | Output the raw server response as JSON |
426
+ | `--json` | no | | Emit machine-readable JSON on stdout |
427
427
 
428
428
  ### `eigenpal workflow dataset example create [options] <workflow-id>`
429
429
 
@@ -446,7 +446,7 @@ Create one eval example without re-uploading the dataset.
446
446
  | `--expected-file <path>` | no | | Expected output from a JSON file (or `-` for stdin) |
447
447
  | `--annotation <text>` | no | | Free-form annotation |
448
448
  | `--base-url <url>` | no | | Server base URL |
449
- | `--json` | no | | Output the raw server response as JSON |
449
+ | `--json` | no | | Emit machine-readable JSON on stdout |
450
450
 
451
451
  ### `eigenpal workflow dataset example update [options] <workflow-id> <exampleId>`
452
452
 
@@ -471,7 +471,7 @@ Patch one eval example. Omitted flags are left alone.
471
471
  | `--annotation <text>` | no | | Replace annotation; pass empty string to clear |
472
472
  | `--row-order <n>` | no | | Reorder the row (0-based) |
473
473
  | `--base-url <url>` | no | | Server base URL |
474
- | `--json` | no | | Output the raw server response as JSON |
474
+ | `--json` | no | | Emit machine-readable JSON on stdout |
475
475
 
476
476
  ### `eigenpal workflow dataset example delete [options] <workflow-id> <exampleId>`
477
477
 
@@ -490,7 +490,7 @@ Delete one eval example by id. Non-TTY shells require --yes.
490
490
  | ------------------ | -------- | ------- | ----------------------------------------------------------------------------- |
491
491
  | `--yes` | no | `false` | Required for non-TTY shells; explicit acknowledgment that this is destructive |
492
492
  | `--base-url <url>` | no | | Server base URL |
493
- | `--json` | no | | Output the raw server response as JSON |
493
+ | `--json` | no | | Emit machine-readable JSON on stdout |
494
494
 
495
495
  ### `eigenpal workflow dataset example get [options] <workflow-id> <exampleId>`
496
496
 
@@ -505,10 +505,10 @@ Fetch one eval example with full triggerInput, expectedOutput, and metadata.
505
505
 
506
506
  ### Options
507
507
 
508
- | Flag | Required | Default | Description |
509
- | ------------------ | -------- | ------- | -------------------------------------- |
510
- | `--base-url <url>` | no | | Server base URL |
511
- | `--json` | no | | Output the raw server response as JSON |
508
+ | Flag | Required | Default | Description |
509
+ | ------------------ | -------- | ------- | ------------------------------------ |
510
+ | `--base-url <url>` | no | | Server base URL |
511
+ | `--json` | no | | Emit machine-readable JSON on stdout |
512
512
 
513
513
  ### `eigenpal workflow dataset validate [options] [path]`
514
514
 
@@ -537,7 +537,7 @@ Upload a DOCX or XLSX file as a new tmpl*… resource with an immutable tmpr*…
537
537
  | `--name <name>` | no | | Display name (defaults to the filename without extension) |
538
538
  | `--description <text>` | no | | Optional description |
539
539
  | `--base-url <url>` | no | | Server base URL |
540
- | `--json` | no | | Output the raw server response as JSON |
540
+ | `--json` | no | | Emit machine-readable JSON on stdout |
541
541
 
542
542
  ### `eigenpal workflow templates list|ls [options]`
543
543
 
@@ -545,12 +545,12 @@ List workspace templates.
545
545
 
546
546
  ### Options
547
547
 
548
- | Flag | Required | Default | Description |
549
- | ------------------ | -------- | ------- | -------------------------------------- |
550
- | `--limit <n>` | no | `50` | Page size |
551
- | `--offset <n>` | no | `0` | Page offset |
552
- | `--base-url <url>` | no | | Server base URL |
553
- | `--json` | no | | Output the raw server response as JSON |
548
+ | Flag | Required | Default | Description |
549
+ | ------------------ | -------- | ------- | ------------------------------------ |
550
+ | `--limit <n>` | no | `50` | Page size |
551
+ | `--offset <n>` | no | `0` | Page offset |
552
+ | `--base-url <url>` | no | | Server base URL |
553
+ | `--json` | no | | Emit machine-readable JSON on stdout |
554
554
 
555
555
  ### `eigenpal workflow templates get|inspect [options] <template-id>`
556
556
 
@@ -564,10 +564,10 @@ Inspect a tmpl*… resource: current tmpr*…, format, checksum, tokens, grammar
564
564
 
565
565
  ### Options
566
566
 
567
- | Flag | Required | Default | Description |
568
- | ------------------ | -------- | ------- | -------------------------------------- |
569
- | `--base-url <url>` | no | | Server base URL |
570
- | `--json` | no | | Output the raw server response as JSON |
567
+ | Flag | Required | Default | Description |
568
+ | ------------------ | -------- | ------- | ------------------------------------ |
569
+ | `--base-url <url>` | no | | Server base URL |
570
+ | `--json` | no | | Emit machine-readable JSON on stdout |
571
571
 
572
572
  ### `eigenpal workflow templates download [options] <template-id>`
573
573
 
@@ -600,10 +600,10 @@ Append an immutable revision and advance the tmpl\_… pointer.
600
600
 
601
601
  ### Options
602
602
 
603
- | Flag | Required | Default | Description |
604
- | ------------------ | -------- | ------- | -------------------------------------- |
605
- | `--base-url <url>` | no | | Server base URL |
606
- | `--json` | no | | Output the raw server response as JSON |
603
+ | Flag | Required | Default | Description |
604
+ | ------------------ | -------- | ------- | ------------------------------------ |
605
+ | `--base-url <url>` | no | | Server base URL |
606
+ | `--json` | no | | Emit machine-readable JSON on stdout |
607
607
 
608
608
  ### `eigenpal workflow templates delete [options] <template-id>`
609
609
 
@@ -617,11 +617,11 @@ Delete the mutable tmpl*… pointer. Pinned tmpr*… revisions remain so workflo
617
617
 
618
618
  ### Options
619
619
 
620
- | Flag | Required | Default | Description |
621
- | ------------------ | -------- | ------- | -------------------------------------- |
622
- | `--yes` | no | `false` | Required for non-TTY shells |
623
- | `--base-url <url>` | no | | Server base URL |
624
- | `--json` | no | | Output the raw server response as JSON |
620
+ | Flag | Required | Default | Description |
621
+ | ------------------ | -------- | ------- | ------------------------------------ |
622
+ | `--yes` | no | `false` | Required for non-TTY shells |
623
+ | `--base-url <url>` | no | | Server base URL |
624
+ | `--json` | no | | Emit machine-readable JSON on stdout |
625
625
 
626
626
  ### `eigenpal workflow templates smoke [options] <template>`
627
627
 
@@ -641,7 +641,7 @@ Fill a local Office file or a tmpl\_… resource with a JSON fixture and write t
641
641
  | `--out <path>` | yes | | Filled DOCX/XLSX output path |
642
642
  | `--revision-id <tmpr>` | no | | When <template> is a tmpl\_… id, pin this revision |
643
643
  | `--base-url <url>` | no | | Server base URL |
644
- | `--json` | no | | Output the raw server response as JSON |
644
+ | `--json` | no | | Emit machine-readable JSON on stdout |
645
645
 
646
646
  ### `eigenpal workflow experiment|exp list|ls [options] <workflow-id>`
647
647
 
@@ -655,13 +655,13 @@ List executions for the workflow, newest first.
655
655
 
656
656
  ### Options
657
657
 
658
- | Flag | Required | Default | Description |
659
- | ------------------ | -------- | ------- | -------------------------------------- |
660
- | `--batch-id <id>` | no | | Filter by batch |
661
- | `--limit <n>` | no | `50` | Page size |
662
- | `--offset <n>` | no | `0` | Page offset |
663
- | `--base-url <url>` | no | | Server base URL |
664
- | `--json` | no | | Output the raw server response as JSON |
658
+ | Flag | Required | Default | Description |
659
+ | ------------------ | -------- | ------- | ------------------------------------ |
660
+ | `--batch-id <id>` | no | | Filter by batch |
661
+ | `--limit <n>` | no | `50` | Page size |
662
+ | `--offset <n>` | no | `0` | Page offset |
663
+ | `--base-url <url>` | no | | Server base URL |
664
+ | `--json` | no | | Emit machine-readable JSON on stdout |
665
665
 
666
666
  ### `eigenpal workflow experiment|exp run [options] <workflow-id>`
667
667
 
@@ -675,13 +675,14 @@ Start a batch eval against the workflow's dataset.
675
675
 
676
676
  ### Options
677
677
 
678
- | Flag | Required | Default | Description |
679
- | ------------------- | -------- | ------- | ---------------------------------------------------- |
680
- | `--example-id <id>` | no | `[]` | Run only this example (repeatable) |
681
- | `--wait` | no | `false` | Poll until terminal; non-zero exit on passRate < 1.0 |
682
- | `--interval <n>` | no | `10` | Polling interval in seconds (default 10) |
683
- | `--base-url <url>` | no | | Server base URL |
684
- | `--json` | no | | Output the raw server response as JSON |
678
+ | Flag | Required | Default | Description |
679
+ | ---------------------- | -------- | ------- | ---------------------------------------------------- |
680
+ | `--example-id <id>` | no | `[]` | Run only this example (repeatable) |
681
+ | `--wait` | no | `false` | Poll until terminal; non-zero exit on passRate < 1.0 |
682
+ | `--interval <n>` | no | `10` | Polling interval in seconds (default 10) |
683
+ | `--max-wait <seconds>` | no | `1800` | Maximum wait before exit code 2 (default 1800) |
684
+ | `--base-url <url>` | no | | Server base URL |
685
+ | `--json` | no | | Emit machine-readable JSON on stdout |
685
686
 
686
687
  ### `eigenpal workflow experiment|exp status [options] <workflow-id> <batchId>`
687
688
 
@@ -704,7 +705,7 @@ Aggregate progress for a batch by `batchId`.
704
705
  | `--max-wait <seconds>` | no | `1800` | Hard ceiling for --watch in seconds (default 1800 = 30 min) |
705
706
  | `--include <kinds>` | no | `""` | Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB) |
706
707
  | `--base-url <url>` | no | | Server base URL |
707
- | `--json` | no | | Output the raw server response as JSON |
708
+ | `--json` | no | | Emit machine-readable JSON on stdout |
708
709
 
709
710
  ### `eigenpal workflow experiment|exp cancel [options] <workflow-id> <batchId>`
710
711
 
@@ -723,7 +724,7 @@ Cancel every execution in a batch. Idempotent.
723
724
  | ------------------ | -------- | ------- | --------------------------------------------------------------------- |
724
725
  | `--yes` | no | | Required for non-TTY shells (CI, pipes). Acts immediately, no prompt. |
725
726
  | `--base-url <url>` | no | | Server base URL |
726
- | `--json` | no | | Output the raw server response as JSON |
727
+ | `--json` | no | | Emit machine-readable JSON on stdout |
727
728
 
728
729
  ### `eigenpal workflow experiment|exp results [options] <workflow-id> [batchId]`
729
730
 
@@ -738,15 +739,20 @@ Download eval results in CSV or JSON.
738
739
 
739
740
  ### Options
740
741
 
741
- | Flag | Required | Default | Description |
742
- | ---------------------- | -------- | ------- | -------------------------------------------------------- |
743
- | `--format <csv\|json>` | yes | | Output format |
744
- | `--out <path>` | no | | Output file. When omitted, the binary streams to stdout. |
745
- | `--base-url <url>` | no | | Server base URL |
742
+ | Flag | Required | Default | Description |
743
+ | ---------------------- | -------- | -------- | ---------------------------------------------------------------------------------------- |
744
+ | `--format <csv\|json>` | no | `"json"` | Output format (default json) |
745
+ | `--out <path>` | no | | Output file. When omitted, the binary streams to stdout. |
746
+ | `--summary` | no | | Show total/pass/fail/error counts, average score, and evaluator rollups |
747
+ | `--failed-only` | no | | Keep only failed or errored evaluator results |
748
+ | `--evaluator <name>` | no | | Keep only results from this evaluator |
749
+ | `--select <path>` | no | | Print only a nested JSON value (for example summary.byEvaluator or discrepancies[].path) |
750
+ | `--base-url <url>` | no | | Server base URL |
751
+ | `--json` | no | | Emit machine-readable JSON on stdout |
746
752
 
747
753
  ### `eigenpal workflow experiment|exp compare|diff [options] <batchIdA> <batchIdB>`
748
754
 
749
- Diff eval scores between two experiment batches.
755
+ Compare evaluator scores or actual outputs between two experiment batches.
750
756
 
751
757
  ### Arguments
752
758
 
@@ -759,10 +765,11 @@ Diff eval scores between two experiment batches.
759
765
 
760
766
  | Flag | Required | Default | Description |
761
767
  | ------------------------------------------------------ | -------- | ------------------ | ------------------------------------------------------ |
768
+ | `--outputs` | no | | Compare actual run outputs instead of evaluator scores |
762
769
  | `--sort <abs-delta-desc\|delta-asc\|delta-desc\|name>` | no | `"abs-delta-desc"` | Row sort order (default: biggest movers first) |
763
770
  | `--regression-threshold <n>` | no | `0.05` | Δ below this is flagged as a regression (default 0.05) |
764
771
  | `--base-url <url>` | no | | Server base URL |
765
- | `--json` | no | | Output the raw server response as JSON |
772
+ | `--json` | no | | Emit machine-readable JSON on stdout |
766
773
 
767
774
  ### `eigenpal workflow experiment|exp watch [options] <workflow-id> <batchId>`
768
775
 
@@ -825,7 +832,7 @@ Create a tagged workflow version from YAML or by copying an existing snapshot.
825
832
  | `--set-version <semver>` | no | | Bare semver tag such as 1.4.0. Required when copying with --from. For --file, omit this flag if the YAML already has a top-level version: field. (Named --set-version to avoid the global -v, --version flag.) |
826
833
  | `--no-activate` | no | | Keep the tagged version off live traffic until you promote it. Default is to make it current. Requires an existing current version. |
827
834
  | `--base-url <url>` | no | | Server base URL |
828
- | `--json` | no | | Output the raw server response as JSON |
835
+ | `--json` | no | | Emit machine-readable JSON on stdout |
829
836
 
830
837
  ### `eigenpal workflow versions promote [options] <workflow-id> <versionId>`
831
838
 
@@ -840,10 +847,10 @@ Make an existing tagged workflow version current without creating another snapsh
840
847
 
841
848
  ### Options
842
849
 
843
- | Flag | Required | Default | Description |
844
- | ------------------ | -------- | ------- | -------------------------------------- |
845
- | `--base-url <url>` | no | | Server base URL |
846
- | `--json` | no | | Output the raw server response as JSON |
850
+ | Flag | Required | Default | Description |
851
+ | ------------------ | -------- | ------- | ------------------------------------ |
852
+ | `--base-url <url>` | no | | Server base URL |
853
+ | `--json` | no | | Emit machine-readable JSON on stdout |
847
854
 
848
855
  ### `eigenpal workflow versions restore [options] <workflow-id> <versionId>`
849
856
 
@@ -862,7 +869,7 @@ Restore a previous snapshot as a new untagged current version. Does not retag th
862
869
  | ------------------ | -------- | ------- | ------------------------------------------------ |
863
870
  | `--message <text>` | no | | Optional restore note stored on the new snapshot |
864
871
  | `--base-url <url>` | no | | Server base URL |
865
- | `--json` | no | | Output the raw server response as JSON |
872
+ | `--json` | no | | Emit machine-readable JSON on stdout |
866
873
 
867
874
  ### `eigenpal workflow step-type list|ls [options]`
868
875
 
@@ -870,13 +877,13 @@ List every step type the deployment supports.
870
877
 
871
878
  ### Options
872
879
 
873
- | Flag | Required | Default | Description |
874
- | ------------------ | -------- | ------- | -------------------------------------- |
875
- | `--search <q>` | no | | Filter |
876
- | `--limit <n>` | no | `50` | Page size |
877
- | `--offset <n>` | no | `0` | Page offset |
878
- | `--base-url <url>` | no | | Server base URL |
879
- | `--json` | no | | Output the raw server response as JSON |
880
+ | Flag | Required | Default | Description |
881
+ | ------------------ | -------- | ------- | ------------------------------------ |
882
+ | `--search <q>` | no | | Filter |
883
+ | `--limit <n>` | no | `50` | Page size |
884
+ | `--offset <n>` | no | `0` | Page offset |
885
+ | `--base-url <url>` | no | | Server base URL |
886
+ | `--json` | no | | Emit machine-readable JSON on stdout |
880
887
 
881
888
  ### `eigenpal workflow step-type get [options] <type>`
882
889
 
@@ -900,13 +907,13 @@ List every evaluator type with a one-line description.
900
907
 
901
908
  ### Options
902
909
 
903
- | Flag | Required | Default | Description |
904
- | ------------------ | -------- | ------- | -------------------------------------- |
905
- | `--search <q>` | no | | Filter by type, name, or description |
906
- | `--limit <n>` | no | `50` | Page size |
907
- | `--offset <n>` | no | `0` | Page offset |
908
- | `--base-url <url>` | no | | Server base URL |
909
- | `--json` | no | | Output the raw server response as JSON |
910
+ | Flag | Required | Default | Description |
911
+ | ------------------ | -------- | ------- | ------------------------------------ |
912
+ | `--search <q>` | no | | Filter by type, name, or description |
913
+ | `--limit <n>` | no | `50` | Page size |
914
+ | `--offset <n>` | no | `0` | Page offset |
915
+ | `--base-url <url>` | no | | Server base URL |
916
+ | `--json` | no | | Emit machine-readable JSON on stdout |
910
917
 
911
918
  ### `eigenpal workflow evaluator-type get [options] <type>`
912
919
 
package/dist/skill/reference/dataset-format.md CHANGED
@@ -142,3 +142,154 @@ For bulk changes, edit the local folder and re-push with
142
142
  `eigenpal workflow dataset example {get,create,update,delete}`. File uploads
143
143
  still go through `dataset push`; CRUD only handles JSON input and
144
144
  `expected.json`-style outputs.
145
+
146
+ <!-- GENERATED:DATASET_REFERENCE START -->
147
+ ## Schema reference
148
+
149
+ _Generated from `@eigenpal/types/src/eval/dataset-archive.ts`, `expected-error.ts`, and `scoped-file-ref.ts`. Do not hand-edit between the GENERATED fences — run `bun run --cwd packages/cli generate:skill`._
150
+
151
+ ### Archive layout (canonical)
152
+
153
+ ```text
154
+ examples/<name>/input.json REQUIRED — full run input object
155
+ examples/<name>/input/<file> OPTIONAL — referenced from input.json
156
+ examples/<name>/expected.json OPTIONAL — success output or `{ "$error": ... }`
157
+ examples/<name>/expected/<file> OPTIONAL — referenced from expected.json
158
+ examples/<name>/meta.json OPTIONAL — see `DatasetMetaSchema` below
159
+ ```
160
+
161
+ Importers reject any archive containing a top-level `manifest.json` (legacy layout). Example folder names must match:
162
+
163
+ `^[a-z0-9][a-z0-9-_]*$`
164
+
165
+ ### `input.json` file references
166
+
167
+ | Field | Type | Required | Default | Description |
168
+ | --- | --- | --- | --- | --- |
169
+ | `$file` | string | yes | | Owner-relative artifact path |
170
+
171
+
172
+ Input files live under `input/` and are referenced as `{ "$file": "input/<path>" }`. Expected files use the `expected/` prefix in `expected.json`.
173
+
174
+ ### `expected.json` — success-expected output
175
+
176
+ When grading success, `expected.json` is a JSON object mirroring the workflow `output:` shape. File values use `{ "$file": "expected/<path>" }`.
177
+
178
+ ### `expected.json` — failure-expected (`$error`)
179
+
180
+ Failure-expected examples store a single top-level `$error` object. At least one of `code`, `messageContains`, or `step` is required:
181
+
182
+ | Field | Type | Required | Default | Description |
183
+ | --- | --- | --- | --- | --- |
184
+ | `code` | integer | no | | |
185
+ | `messageContains` | string | no | | |
186
+ | `step` | string | no | | |
187
+
188
+
189
+ ### `meta.json` (`DatasetMetaSchema`)
190
+
191
+ | Field | Type | Required | Default | Description |
192
+ | --- | --- | --- | --- | --- |
193
+ | `rowOrder` | integer | no | | Non-negative display order hint for this example. |
194
+ | `annotation` | string | no | | Free-form example note, limited to 2000 characters. |
195
+ | `overrides` | record<string, unknown> | no | | Per-step output overrides as `{ "steps": { "<stepName>": <outputObject> } }`; overridden steps are skipped or partially merged during evaluation. |
196
+
197
+
198
+ ### Import mode (`DatasetImportModeSchema`)
199
+
200
+ Allowed values: `"append"` \| `"replace"`
201
+
202
+ - `append` — add examples to the existing dataset.
203
+ - `replace` — wipe existing examples and import the archive fresh.
204
+
205
+ ### Validation rules enforced at import
206
+
207
+ - Every example requires `input.json` as a JSON object (may be `{}`).
208
+ - Files under `input/` must be referenced from `input.json`; unreferenced files are rejected.
209
+ - Files under `expected/` must be referenced from `expected.json` when present.
210
+ - File reference paths cannot use `..`, absolute paths, backslashes, or null bytes.
211
+ - `$error` examples are supported for workflow datasets only; agent datasets reject them.
212
+ - Archives larger than 500 MB are rejected at import.
213
+
214
+ ### Complete machine-readable component schemas
215
+
216
+ `meta.json`:
217
+
218
+ ```json
219
+ {
220
+ "$schema": "http://json-schema.org/draft-07/schema#",
221
+ "type": "object",
222
+ "properties": {
223
+ "rowOrder": {
224
+ "description": "Non-negative display order hint for this example.",
225
+ "type": "integer",
226
+ "minimum": 0,
227
+ "maximum": 9007199254740991
228
+ },
229
+ "annotation": {
230
+ "description": "Free-form example note, limited to 2000 characters.",
231
+ "type": "string",
232
+ "maxLength": 2000
233
+ },
234
+ "overrides": {
235
+ "description": "Per-step output overrides as `{ \"steps\": { \"<stepName>\": <outputObject> } }`; overridden steps are skipped or partially merged during evaluation.",
236
+ "type": "object",
237
+ "propertyNames": {
238
+ "type": "string"
239
+ },
240
+ "additionalProperties": {}
241
+ }
242
+ },
243
+ "additionalProperties": false
244
+ }
245
+ ```
246
+
247
+
248
+ Failure-expected `$error` object:
249
+
250
+ ```json
251
+ {
252
+ "$schema": "http://json-schema.org/draft-07/schema#",
253
+ "type": "object",
254
+ "properties": {
255
+ "code": {
256
+ "type": "integer",
257
+ "minimum": 400,
258
+ "maximum": 599
259
+ },
260
+ "messageContains": {
261
+ "type": "string",
262
+ "minLength": 1,
263
+ "maxLength": 1000
264
+ },
265
+ "step": {
266
+ "type": "string",
267
+ "minLength": 1,
268
+ "maxLength": 200
269
+ }
270
+ },
271
+ "additionalProperties": false
272
+ }
273
+ ```
274
+
275
+
276
+ Scoped `$file` reference:
277
+
278
+ ```json
279
+ {
280
+ "$schema": "http://json-schema.org/draft-07/schema#",
281
+ "type": "object",
282
+ "properties": {
283
+ "$file": {
284
+ "type": "string",
285
+ "minLength": 1,
286
+ "description": "Owner-relative artifact path"
287
+ }
288
+ },
289
+ "required": [
290
+ "$file"
291
+ ],
292
+ "additionalProperties": false
293
+ }
294
+ ```
295
+ <!-- GENERATED:DATASET_REFERENCE END -->
package/dist/skill/reference/debugging.md CHANGED
@@ -95,6 +95,7 @@ Drill into one field:
95
95
 
96
96
  ```bash
97
97
  eigenpal runs get <exec-id> --expand execution --json | jq '.stepExecutions[2].output.totalAmount'
98
+ eigenpal runs get <exec-id> --select output.subjects
98
99
  ```
99
100
 
100
101
  ## 3. Common failure modes
@@ -231,10 +232,18 @@ Behavior by current state:
231
232
  eigenpal workflow experiment compare <batch-a> <batch-b>
232
233
  eigenpal workflow experiment compare <batch-a> <batch-b> --regression-threshold 0.10
233
234
  eigenpal workflow experiment compare <batch-a> <batch-b> --json | jq '.summary.regressions'
235
+ eigenpal workflow experiment compare <batch-a> <batch-b> --outputs
236
+ eigenpal workflow experiment results <workflow-id> <batch-id> --summary
237
+ eigenpal workflow experiment results <workflow-id> <batch-id> --failed-only --evaluator fields-match
238
+ eigenpal agents experiment compare <batch-a> <batch-b> --outputs
239
+ eigenpal agents experiment results <agent-id-or-slug> <batch-id> --summary
234
240
  ```
235
241
 
236
242
  Side-by-side score diff per `(example, evaluator)` pair. Highlights
237
243
  regressions vs improvements; aggregate stats at the bottom.
244
+ Use `--outputs` to compare the actual JSON outputs by example instead of
245
+ evaluator scores. `experiment results --summary` reports run and evaluator
246
+ rollups; `--failed-only` and `--evaluator` narrow the structured discrepancies.
238
247
 
239
248
  - `--sort abs-delta-desc` (default — biggest movers first) | `delta-asc`
240
249
  (regressions first) | `delta-desc` (improvements first) | `name`