@eigenpal/cli 0.19.1 → 0.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (9) hide show
  1. package/CHANGELOG.md +6 -0
  2. package/dist/cli.js +425 -9
  3. package/dist/docs-bundle.json +17 -10
  4. package/dist/skill/SKILL.md +32 -0
  5. package/dist/skill/reference/cli/agents.md +181 -7
  6. package/dist/skill/reference/cli/surface.md +18 -2
  7. package/dist/skill/reference/cli/workflow.md +185 -11
  8. package/dist/skill/reference/dataset-format.md +55 -0
  9. package/package.json +1 -1
package/dist/skill/SKILL.md CHANGED
@@ -164,6 +164,30 @@ eigenpal workflow dataset push <workflow-id> --file ./dataset --mode replace
164
164
  eigenpal workflow evaluators validate ./evaluators.yaml
165
165
  eigenpal workflow evaluators push <workflow-id> --file ./evaluators.yaml
166
166
 
167
+ # If you were given any examples: request a dataset review BEFORE iterating
168
+ # to perfection. Ship a V0, then get ground truth reviewed first — a quick
169
+ # unreviewed V0 is fine, but never finalize evaluators, experiments, or
170
+ # versions against unreviewed expected outputs.
171
+ eigenpal workflow dataset review-request create <workflow-id> \
172
+ --title "<topic> GT review" --example-name <example-name> \
173
+ --focus <path> --focus-reason '<path>=why' --ignore <path> \
174
+ --item-note '<example>=note' --field-note '<example.path>=note' \
175
+ --status open --json
176
+ # Poll .progress.complete until reviewers finish. Record field decisions and
177
+ # approve/reject with comments as needed, then pull and MANUAL reconcile —
178
+ # there is no auto-apply by design (reviewers can err). Close when done.
179
+ eigenpal workflow dataset review-request get <workflow-id> <review-id> --json | jq '.progress'
180
+ eigenpal workflow dataset review-request item <workflow-id> <review-id> <item-id> \
181
+ --action field-decision --field-path <path> --decision approved \
182
+ --expected-updated-at <iso> --json
183
+ eigenpal workflow dataset review-request item <workflow-id> <review-id> <item-id> \
184
+ --action reject --comment "not usable" --expected-updated-at <iso> --json
185
+ eigenpal workflow dataset pull <workflow-id> --out ./dataset.zip
186
+ # Manually copy approved/edited expected.json per example into the dataset,
187
+ # then push. Reject is recommendation-only — nothing is deleted by the API.
188
+ eigenpal workflow dataset review-request update <workflow-id> <review-id> \
189
+ --status closed --json
190
+
167
191
  # Run an experiment.
168
192
  eigenpal workflow experiment run <workflow-id>
169
193
  eigenpal workflow experiment watch <workflow-id> <batch-id>
@@ -379,6 +403,14 @@ eigenpal agents dataset pull agents.<slug> --out ./dataset.zip
379
403
  Expected outputs should include stable fields only. Omit timestamps, random IDs,
380
404
  and open-ended LLM text.
381
405
 
406
+ If you were given any examples: request a dataset review BEFORE iterating
407
+ to perfection (`eigenpal agents dataset review-request create agents.<slug>
408
+ --title ... --example-name ... --focus ... --status open --json`). Ship a V0,
409
+ get ground truth reviewed first — poll `.progress.complete`, record field
410
+ decisions / approve / reject with comments, `dataset pull` and MANUAL
411
+ per-example reconcile (no auto-apply), then
412
+ `review-request update ... --status closed`.
413
+
382
414
  ## Runs And Artifacts
383
415
 
384
416
  ```bash
package/dist/skill/reference/cli/agents.md CHANGED
@@ -45,6 +45,13 @@ Manage Eigenpal agents: Git source, datasets, experiments, sessions, and release
45
45
  - [`eigenpal agents dataset push [options] <agent-id-or-slug>`](#eigenpal-agents-dataset-push-options-agent-id-or-slug)
46
46
  - [`eigenpal agents dataset pull [options] <agent-id-or-slug>`](#eigenpal-agents-dataset-pull-options-agent-id-or-slug)
47
47
  - [`eigenpal agents dataset validate [options] [path]`](#eigenpal-agents-dataset-validate-options-path)
48
+ - [`eigenpal agents dataset review-request list|ls [options] <automation-id>`](#eigenpal-agents-dataset-review-request-listls-options-automation-id)
49
+ - [`eigenpal agents dataset review-request create [options] <automation-id>`](#eigenpal-agents-dataset-review-request-create-options-automation-id)
50
+ - [`eigenpal agents dataset review-request get [options] <automation-id> <review-id>`](#eigenpal-agents-dataset-review-request-get-options-automation-id-review-id)
51
+ - [`eigenpal agents dataset review-request update [options] <automation-id> <review-id>`](#eigenpal-agents-dataset-review-request-update-options-automation-id-review-id)
52
+ - [`eigenpal agents dataset review-request items [options] <automation-id> <review-id>`](#eigenpal-agents-dataset-review-request-items-options-automation-id-review-id)
53
+ - [`eigenpal agents dataset review-request events [options] <automation-id> <review-id>`](#eigenpal-agents-dataset-review-request-events-options-automation-id-review-id)
54
+ - [`eigenpal agents dataset review-request item [options] <automation-id> <review-id> <item-id>`](#eigenpal-agents-dataset-review-request-item-options-automation-id-review-id-item-id)
48
55
  - [`eigenpal agents experiment|exp run [options] <agent-id-or-slug>`](#eigenpal-agents-experimentexp-run-options-agent-id-or-slug)
49
56
  - [`eigenpal agents experiment|exp status [options] <agent-id-or-slug> <batch-id>`](#eigenpal-agents-experimentexp-status-options-agent-id-or-slug-batch-id)
50
57
  - [`eigenpal agents experiment|exp results [options] <agent-id-or-slug> [batch-id]`](#eigenpal-agents-experimentexp-results-options-agent-id-or-slug-batch-id)
@@ -95,7 +102,15 @@ agents
95
102
  │ ├── list|ls <agent-id-or-slug>
96
103
  │ ├── push <agent-id-or-slug>
97
104
  │ ├── pull <agent-id-or-slug>
98
- │ └── validate [path]
105
+ │ ├── validate [path]
106
+ │ └── review-request
107
+ │ ├── list|ls <automation-id>
108
+ │ ├── create <automation-id>
109
+ │ ├── get <automation-id> <review-id>
110
+ │ ├── update <automation-id> <review-id>
111
+ │ ├── items <automation-id> <review-id>
112
+ │ ├── events <automation-id> <review-id>
113
+ │ └── item <automation-id> <review-id> <item-id>
99
114
  ├── experiment|exp
100
115
  │ ├── run <agent-id-or-slug>
101
116
  │ ├── status <agent-id-or-slug> <batch-id>
@@ -159,12 +174,19 @@ agents
159
174
 
160
175
  ### Dataset
161
176
 
162
- | Command | Description |
163
- | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
164
- | `eigenpal agents dataset list\|ls [options] <agent-id-or-slug>` | List dataset examples for an agent. |
165
- | `eigenpal agents dataset push [options] <agent-id-or-slug>` | Upload dataset examples from a local dataset directory or zip archive. |
166
- | `eigenpal agents dataset pull [options] <agent-id-or-slug>` | Download an agent dataset as a .zip archive. |
167
- | `eigenpal agents dataset validate [options] [path]` | Validate a local dataset directory against the canonical examples/<name> layout. Defaults to ./dataset/. |
177
+ | Command | Description |
178
+ | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
179
+ | `eigenpal agents dataset list\|ls [options] <agent-id-or-slug>` | List dataset examples for an agent. |
180
+ | `eigenpal agents dataset push [options] <agent-id-or-slug>` | Upload dataset examples from a local dataset directory or zip archive. |
181
+ | `eigenpal agents dataset pull [options] <agent-id-or-slug>` | Download an agent dataset as a .zip archive. |
182
+ | `eigenpal agents dataset validate [options] [path]` | Validate a local dataset directory against the canonical examples/<name> layout. Defaults to ./dataset/. |
183
+ | `eigenpal agents dataset review-request list\|ls [options] <automation-id>` | |
184
+ | `eigenpal agents dataset review-request create [options] <automation-id>` | Request human review of dataset ground truth. Snapshot examples and poll progress until review is complete. |
185
+ | `eigenpal agents dataset review-request get [options] <automation-id> <review-id>` | Fetch one dataset review request with items, progress, focus fields, ignored fields, and events. |
186
+ | `eigenpal agents dataset review-request update [options] <automation-id> <review-id>` | Update review metadata or lifecycle status. Set --status closed when review is finished. |
187
+ | `eigenpal agents dataset review-request items [options] <automation-id> <review-id>` | List snapshotted review items and their statuses. |
188
+ | `eigenpal agents dataset review-request events [options] <automation-id> <review-id>` | List review activity, including example and field notes. |
189
+ | `eigenpal agents dataset review-request item [options] <automation-id> <review-id> <item-id>` | Approve, reject, reopen, comment, edit, or record a field-decision on one review item. |
168
190
 
169
191
  ### Experiment
170
192
 
@@ -670,6 +692,158 @@ Validate a local dataset directory against the canonical examples/<name> layout.
670
692
  | `--json` | no | | Emit machine-readable JSON on stdout |
671
693
  | `--agent-dir <dir>` | no | `"."` | Agent package directory with optional input/output schemas for extra value-level checks |
672
694
 
695
+ ### `eigenpal agents dataset review-request list|ls [options] <automation-id>`
696
+
697
+ ### Arguments
698
+
699
+ | Name | Required | Variadic | Description |
700
+ | --------------- | -------- | -------- | ----------- |
701
+ | `automation-id` | yes | no | |
702
+
703
+ ### Options
704
+
705
+ | Flag | Required | Default | Description |
706
+ | ------------------ | -------- | ------- | ------------------------------------------------------------------- |
707
+ | `--base-url <url>` | no | | Server base URL |
708
+ | `--status <csv>` | no | | Filter by review status (comma-separated: draft,open,paused,closed) |
709
+ | `--limit <n>` | no | `50` | Page size |
710
+ | `--offset <n>` | no | `0` | Page offset |
711
+ | `--json` | no | | Emit machine-readable JSON on stdout |
712
+
713
+ ### `eigenpal agents dataset review-request create [options] <automation-id>`
714
+
715
+ Request human review of dataset ground truth. Snapshot examples and poll progress until review is complete.
716
+
717
+ ### Arguments
718
+
719
+ | Name | Required | Variadic | Description |
720
+ | --------------- | -------- | -------- | ----------- |
721
+ | `automation-id` | yes | no | |
722
+
723
+ ### Options
724
+
725
+ | Flag | Required | Default | Description |
726
+ | -------------------------------- | -------- | ------- | -------------------------------------------------------------------------------- |
727
+ | `--base-url <url>` | no | | Server base URL |
728
+ | `--json` | no | | Emit machine-readable JSON on stdout |
729
+ | `--title <title>` | yes | | Review request title |
730
+ | `--example-name <name>` | yes | `[]` | Example folder name to include (repeatable) |
731
+ | `--instructions <text>` | no | | Note shown to the reviewer for the whole request |
732
+ | `--focus <path>` | no | `[]` | Expected-output path to highlight (repeatable) |
733
+ | `--focus-reason <spec>` | no | `[]` | Reason a focus path needs review, as path=reason (repeatable) |
734
+ | `--ignore <path>` | no | `[]` | Expected-output path reviewers can skip (repeatable) |
735
+ | `--item-note <spec>` | no | `[]` | Example-level note seeded at create time, as exampleName=comment (repeatable) |
736
+ | `--field-note <spec>` | no | `[]` | Field-level note seeded at create time, as exampleName.path=comment (repeatable) |
737
+ | `--focus-json <json>` | no | | JSON array of { path, reason? } focus fields |
738
+ | `--notes-json <json>` | no | | JSON array of { exampleName, comment?, fields?: [{ path, comment }] } |
739
+ | `--status <draft\|open\|paused>` | no | | Initial lifecycle status (default: draft) |
740
+
741
+ ### `eigenpal agents dataset review-request get [options] <automation-id> <review-id>`
742
+
743
+ Fetch one dataset review request with items, progress, focus fields, ignored fields, and events.
744
+
745
+ ### Arguments
746
+
747
+ | Name | Required | Variadic | Description |
748
+ | --------------- | -------- | -------- | ----------- |
749
+ | `automation-id` | yes | no | |
750
+ | `review-id` | yes | no | |
751
+
752
+ ### Options
753
+
754
+ | Flag | Required | Default | Description |
755
+ | ------------------ | -------- | ------- | ------------------------------------ |
756
+ | `--base-url <url>` | no | | Server base URL |
757
+ | `--json` | no | | Emit machine-readable JSON on stdout |
758
+
759
+ ### `eigenpal agents dataset review-request update [options] <automation-id> <review-id>`
760
+
761
+ Update review metadata or lifecycle status. Set --status closed when review is finished.
762
+
763
+ ### Arguments
764
+
765
+ | Name | Required | Variadic | Description |
766
+ | --------------- | -------- | -------- | ----------- |
767
+ | `automation-id` | yes | no | |
768
+ | `review-id` | yes | no | |
769
+
770
+ ### Options
771
+
772
+ | Flag | Required | Default | Description |
773
+ | ---------------------------------------- | -------- | ------- | ------------------------------------------------------------------------ |
774
+ | `--base-url <url>` | no | | Server base URL |
775
+ | `--json` | no | | Emit machine-readable JSON on stdout |
776
+ | `--title <title>` | no | | New review request title |
777
+ | `--instructions <text>` | no | | Note shown to the reviewer for the whole request |
778
+ | `--focus <path>` | no | `[]` | Replace focus paths (repeatable; use with --focus-reason / --focus-json) |
779
+ | `--focus-reason <spec>` | no | `[]` | Reason a focus path needs review, as path=reason (repeatable) |
780
+ | `--focus-json <json>` | no | | JSON array of { path, reason? } focus fields (replaces focus) |
781
+ | `--ignore <path>` | no | | Replace ignored paths (repeatable) |
782
+ | `--status <draft\|open\|paused\|closed>` | no | | Lifecycle status |
783
+
784
+ ### `eigenpal agents dataset review-request items [options] <automation-id> <review-id>`
785
+
786
+ List snapshotted review items and their statuses.
787
+
788
+ ### Arguments
789
+
790
+ | Name | Required | Variadic | Description |
791
+ | --------------- | -------- | -------- | ----------- |
792
+ | `automation-id` | yes | no | |
793
+ | `review-id` | yes | no | |
794
+
795
+ ### Options
796
+
797
+ | Flag | Required | Default | Description |
798
+ | ------------------ | -------- | ------- | -------------------------------------------------------- |
799
+ | `--base-url <url>` | no | | Server base URL |
800
+ | `--json` | no | | Emit machine-readable JSON on stdout |
801
+ | `--status <csv>` | no | | Filter by item status (pending,approved,edited,rejected) |
802
+
803
+ ### `eigenpal agents dataset review-request events [options] <automation-id> <review-id>`
804
+
805
+ List review activity, including example and field notes.
806
+
807
+ ### Arguments
808
+
809
+ | Name | Required | Variadic | Description |
810
+ | --------------- | -------- | -------- | ----------- |
811
+ | `automation-id` | yes | no | |
812
+ | `review-id` | yes | no | |
813
+
814
+ ### Options
815
+
816
+ | Flag | Required | Default | Description |
817
+ | ------------------ | -------- | ------- | ------------------------------------ |
818
+ | `--base-url <url>` | no | | Server base URL |
819
+ | `--json` | no | | Emit machine-readable JSON on stdout |
820
+
821
+ ### `eigenpal agents dataset review-request item [options] <automation-id> <review-id> <item-id>`
822
+
823
+ Approve, reject, reopen, comment, edit, or record a field-decision on one review item.
824
+
825
+ ### Arguments
826
+
827
+ | Name | Required | Variadic | Description |
828
+ | --------------- | -------- | -------- | ----------- |
829
+ | `automation-id` | yes | no | |
830
+ | `review-id` | yes | no | |
831
+ | `item-id` | yes | no | |
832
+
833
+ ### Options
834
+
835
+ | Flag | Required | Default | Description |
836
+ | ------------------------------------------------------------------- | -------- | ------- | ------------------------------------------------------------------ |
837
+ | `--base-url <url>` | no | | Server base URL |
838
+ | `--json` | no | | Emit machine-readable JSON on stdout |
839
+ | `--action <approve\|reject\|reopen\|comment\|edit\|field-decision>` | yes | | Item action |
840
+ | `--expected-updated-at <iso>` | yes | | Item updatedAt the client last observed (optimistic concurrency) |
841
+ | `--comment <text>` | no | | Note stored on the item or field |
842
+ | `--field-path <path>` | no | | Dotted expected-output path for comment or field-decision |
843
+ | `--decision <approved\|rejected\|null>` | no | | Field decision for --action field-decision (null/clear removes it) |
844
+ | `--clear` | no | `false` | Clear a field decision (sends decision: null) |
845
+ | `--expected-json <json>` | no | | Replacement expected JSON when --action edit |
846
+
673
847
  ### `eigenpal agents experiment|exp run [options] <agent-id-or-slug>`
674
848
 
675
849
  Start an experiment over dataset examples.
package/dist/skill/reference/cli/surface.md CHANGED
@@ -37,7 +37,15 @@ eigenpal
37
37
  │ │ │ ├── update <workflow-id> <exampleId>
38
38
  │ │ │ ├── delete <workflow-id> <exampleId>
39
39
  │ │ │ └── get <workflow-id> <exampleId>
40
- │ │ └── validate [path]
40
+ │ │ ├── validate [path]
41
+ │ │ └── review-request
42
+ │ │ ├── list|ls <automation-id>
43
+ │ │ ├── create <automation-id>
44
+ │ │ ├── get <automation-id> <review-id>
45
+ │ │ ├── update <automation-id> <review-id>
46
+ │ │ ├── items <automation-id> <review-id>
47
+ │ │ ├── events <automation-id> <review-id>
48
+ │ │ └── item <automation-id> <review-id> <item-id>
41
49
  │ ├── templates
42
50
  │ │ ├── upload <file>
43
51
  │ │ ├── list|ls
@@ -103,7 +111,15 @@ eigenpal
103
111
  │ │ ├── list|ls <agent-id-or-slug>
104
112
  │ │ ├── push <agent-id-or-slug>
105
113
  │ │ ├── pull <agent-id-or-slug>
106
- │ │ └── validate [path]
114
+ │ │ ├── validate [path]
115
+ │ │ └── review-request
116
+ │ │ ├── list|ls <automation-id>
117
+ │ │ ├── create <automation-id>
118
+ │ │ ├── get <automation-id> <review-id>
119
+ │ │ ├── update <automation-id> <review-id>
120
+ │ │ ├── items <automation-id> <review-id>
121
+ │ │ ├── events <automation-id> <review-id>
122
+ │ │ └── item <automation-id> <review-id> <item-id>
107
123
  │ ├── experiment|exp
108
124
  │ │ ├── run <agent-id-or-slug>
109
125
  │ │ ├── status <agent-id-or-slug> <batch-id>
package/dist/skill/reference/cli/workflow.md CHANGED
@@ -40,6 +40,13 @@ Manage workflows: push, pull, and evaluate.
40
40
  - [`eigenpal workflow dataset example delete [options] <workflow-id> <exampleId>`](#eigenpal-workflow-dataset-example-delete-options-workflow-id-exampleid)
41
41
  - [`eigenpal workflow dataset example get [options] <workflow-id> <exampleId>`](#eigenpal-workflow-dataset-example-get-options-workflow-id-exampleid)
42
42
  - [`eigenpal workflow dataset validate [options] [path]`](#eigenpal-workflow-dataset-validate-options-path)
43
+ - [`eigenpal workflow dataset review-request list|ls [options] <automation-id>`](#eigenpal-workflow-dataset-review-request-listls-options-automation-id)
44
+ - [`eigenpal workflow dataset review-request create [options] <automation-id>`](#eigenpal-workflow-dataset-review-request-create-options-automation-id)
45
+ - [`eigenpal workflow dataset review-request get [options] <automation-id> <review-id>`](#eigenpal-workflow-dataset-review-request-get-options-automation-id-review-id)
46
+ - [`eigenpal workflow dataset review-request update [options] <automation-id> <review-id>`](#eigenpal-workflow-dataset-review-request-update-options-automation-id-review-id)
47
+ - [`eigenpal workflow dataset review-request items [options] <automation-id> <review-id>`](#eigenpal-workflow-dataset-review-request-items-options-automation-id-review-id)
48
+ - [`eigenpal workflow dataset review-request events [options] <automation-id> <review-id>`](#eigenpal-workflow-dataset-review-request-events-options-automation-id-review-id)
49
+ - [`eigenpal workflow dataset review-request item [options] <automation-id> <review-id> <item-id>`](#eigenpal-workflow-dataset-review-request-item-options-automation-id-review-id-item-id)
43
50
  - [`eigenpal workflow templates upload [options] <file>`](#eigenpal-workflow-templates-upload-options-file)
44
51
  - [`eigenpal workflow templates list|ls [options]`](#eigenpal-workflow-templates-listls-options)
45
52
  - [`eigenpal workflow templates get|inspect [options] <template-id>`](#eigenpal-workflow-templates-getinspect-options-template-id)
@@ -92,7 +99,15 @@ workflow
92
99
  │ │ ├── update <workflow-id> <exampleId>
93
100
  │ │ ├── delete <workflow-id> <exampleId>
94
101
  │ │ └── get <workflow-id> <exampleId>
95
- │ └── validate [path]
102
+ │ ├── validate [path]
103
+ │ └── review-request
104
+ │ ├── list|ls <automation-id>
105
+ │ ├── create <automation-id>
106
+ │ ├── get <automation-id> <review-id>
107
+ │ ├── update <automation-id> <review-id>
108
+ │ ├── items <automation-id> <review-id>
109
+ │ ├── events <automation-id> <review-id>
110
+ │ └── item <automation-id> <review-id> <item-id>
96
111
  ├── templates
97
112
  │ ├── upload <file>
98
113
  │ ├── list|ls
@@ -160,16 +175,23 @@ workflow
160
175
 
161
176
  ### Dataset
162
177
 
163
- | Command | Description |
164
- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
165
- | `eigenpal workflow dataset list\|ls [options] <workflow-id>` | List eval examples for the workflow. |
166
- | `eigenpal workflow dataset pull [options] <workflow-id>` | Download the workflow's dataset as a ZIP archive. |
167
- | `eigenpal workflow dataset push [options] <workflow-id>` | Replace or extend the workflow's dataset from a ZIP or folder. |
168
- | `eigenpal workflow dataset example create [options] <workflow-id>` | Create one eval example without re-uploading the dataset. |
169
- | `eigenpal workflow dataset example update [options] <workflow-id> <exampleId>` | Patch one eval example. Omitted flags are left alone. |
170
- | `eigenpal workflow dataset example delete [options] <workflow-id> <exampleId>` | Delete one eval example by id. Non-TTY shells require --yes. |
171
- | `eigenpal workflow dataset example get [options] <workflow-id> <exampleId>` | Fetch one eval example with full triggerInput, expectedOutput, and metadata. |
172
- | `eigenpal workflow dataset validate [options] [path]` | Validate a dataset folder against the examples/<name>/{input,expected,meta} convention. Defaults to ./dataset/. |
178
+ | Command | Description |
179
+ | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
180
+ | `eigenpal workflow dataset list\|ls [options] <workflow-id>` | List eval examples for the workflow. |
181
+ | `eigenpal workflow dataset pull [options] <workflow-id>` | Download the workflow's dataset as a ZIP archive. |
182
+ | `eigenpal workflow dataset push [options] <workflow-id>` | Replace or extend the workflow's dataset from a ZIP or folder. |
183
+ | `eigenpal workflow dataset example create [options] <workflow-id>` | Create one eval example without re-uploading the dataset. |
184
+ | `eigenpal workflow dataset example update [options] <workflow-id> <exampleId>` | Patch one eval example. Omitted flags are left alone. |
185
+ | `eigenpal workflow dataset example delete [options] <workflow-id> <exampleId>` | Delete one eval example by id. Non-TTY shells require --yes. |
186
+ | `eigenpal workflow dataset example get [options] <workflow-id> <exampleId>` | Fetch one eval example with full triggerInput, expectedOutput, and metadata. |
187
+ | `eigenpal workflow dataset validate [options] [path]` | Validate a dataset folder against the examples/<name>/{input,expected,meta} convention. Defaults to ./dataset/. |
188
+ | `eigenpal workflow dataset review-request list\|ls [options] <automation-id>` | |
189
+ | `eigenpal workflow dataset review-request create [options] <automation-id>` | Request human review of dataset ground truth. Snapshot examples and poll progress until review is complete. |
190
+ | `eigenpal workflow dataset review-request get [options] <automation-id> <review-id>` | Fetch one dataset review request with items, progress, focus fields, ignored fields, and events. |
191
+ | `eigenpal workflow dataset review-request update [options] <automation-id> <review-id>` | Update review metadata or lifecycle status. Set --status closed when review is finished. |
192
+ | `eigenpal workflow dataset review-request items [options] <automation-id> <review-id>` | List snapshotted review items and their statuses. |
193
+ | `eigenpal workflow dataset review-request events [options] <automation-id> <review-id>` | List review activity, including example and field notes. |
194
+ | `eigenpal workflow dataset review-request item [options] <automation-id> <review-id> <item-id>` | Approve, reject, reopen, comment, edit, or record a field-decision on one review item. |
173
195
 
174
196
  ### Templates
175
197
 
@@ -625,6 +647,158 @@ Validate a dataset folder against the examples/<name>/{input,expected,meta} conv
625
647
  | ------ | -------- | -------- | ----------- |
626
648
  | `path` | no | no | |
627
649
 
650
+ ### `eigenpal workflow dataset review-request list|ls [options] <automation-id>`
651
+
652
+ ### Arguments
653
+
654
+ | Name | Required | Variadic | Description |
655
+ | --------------- | -------- | -------- | ----------- |
656
+ | `automation-id` | yes | no | |
657
+
658
+ ### Options
659
+
660
+ | Flag | Required | Default | Description |
661
+ | ------------------ | -------- | ------- | ------------------------------------------------------------------- |
662
+ | `--base-url <url>` | no | | Server base URL |
663
+ | `--status <csv>` | no | | Filter by review status (comma-separated: draft,open,paused,closed) |
664
+ | `--limit <n>` | no | `50` | Page size |
665
+ | `--offset <n>` | no | `0` | Page offset |
666
+ | `--json` | no | | Emit machine-readable JSON on stdout |
667
+
668
+ ### `eigenpal workflow dataset review-request create [options] <automation-id>`
669
+
670
+ Request human review of dataset ground truth. Snapshot examples and poll progress until review is complete.
671
+
672
+ ### Arguments
673
+
674
+ | Name | Required | Variadic | Description |
675
+ | --------------- | -------- | -------- | ----------- |
676
+ | `automation-id` | yes | no | |
677
+
678
+ ### Options
679
+
680
+ | Flag | Required | Default | Description |
681
+ | -------------------------------- | -------- | ------- | -------------------------------------------------------------------------------- |
682
+ | `--base-url <url>` | no | | Server base URL |
683
+ | `--json` | no | | Emit machine-readable JSON on stdout |
684
+ | `--title <title>` | yes | | Review request title |
685
+ | `--example-name <name>` | yes | `[]` | Example folder name to include (repeatable) |
686
+ | `--instructions <text>` | no | | Note shown to the reviewer for the whole request |
687
+ | `--focus <path>` | no | `[]` | Expected-output path to highlight (repeatable) |
688
+ | `--focus-reason <spec>` | no | `[]` | Reason a focus path needs review, as path=reason (repeatable) |
689
+ | `--ignore <path>` | no | `[]` | Expected-output path reviewers can skip (repeatable) |
690
+ | `--item-note <spec>` | no | `[]` | Example-level note seeded at create time, as exampleName=comment (repeatable) |
691
+ | `--field-note <spec>` | no | `[]` | Field-level note seeded at create time, as exampleName.path=comment (repeatable) |
692
+ | `--focus-json <json>` | no | | JSON array of { path, reason? } focus fields |
693
+ | `--notes-json <json>` | no | | JSON array of { exampleName, comment?, fields?: [{ path, comment }] } |
694
+ | `--status <draft\|open\|paused>` | no | | Initial lifecycle status (default: draft) |
695
+
696
+ ### `eigenpal workflow dataset review-request get [options] <automation-id> <review-id>`
697
+
698
+ Fetch one dataset review request with items, progress, focus fields, ignored fields, and events.
699
+
700
+ ### Arguments
701
+
702
+ | Name | Required | Variadic | Description |
703
+ | --------------- | -------- | -------- | ----------- |
704
+ | `automation-id` | yes | no | |
705
+ | `review-id` | yes | no | |
706
+
707
+ ### Options
708
+
709
+ | Flag | Required | Default | Description |
710
+ | ------------------ | -------- | ------- | ------------------------------------ |
711
+ | `--base-url <url>` | no | | Server base URL |
712
+ | `--json` | no | | Emit machine-readable JSON on stdout |
713
+
714
+ ### `eigenpal workflow dataset review-request update [options] <automation-id> <review-id>`
715
+
716
+ Update review metadata or lifecycle status. Set --status closed when review is finished.
717
+
718
+ ### Arguments
719
+
720
+ | Name | Required | Variadic | Description |
721
+ | --------------- | -------- | -------- | ----------- |
722
+ | `automation-id` | yes | no | |
723
+ | `review-id` | yes | no | |
724
+
725
+ ### Options
726
+
727
+ | Flag | Required | Default | Description |
728
+ | ---------------------------------------- | -------- | ------- | ------------------------------------------------------------------------ |
729
+ | `--base-url <url>` | no | | Server base URL |
730
+ | `--json` | no | | Emit machine-readable JSON on stdout |
731
+ | `--title <title>` | no | | New review request title |
732
+ | `--instructions <text>` | no | | Note shown to the reviewer for the whole request |
733
+ | `--focus <path>` | no | `[]` | Replace focus paths (repeatable; use with --focus-reason / --focus-json) |
734
+ | `--focus-reason <spec>` | no | `[]` | Reason a focus path needs review, as path=reason (repeatable) |
735
+ | `--focus-json <json>` | no | | JSON array of { path, reason? } focus fields (replaces focus) |
736
+ | `--ignore <path>` | no | | Replace ignored paths (repeatable) |
737
+ | `--status <draft\|open\|paused\|closed>` | no | | Lifecycle status |
738
+
739
+ ### `eigenpal workflow dataset review-request items [options] <automation-id> <review-id>`
740
+
741
+ List snapshotted review items and their statuses.
742
+
743
+ ### Arguments
744
+
745
+ | Name | Required | Variadic | Description |
746
+ | --------------- | -------- | -------- | ----------- |
747
+ | `automation-id` | yes | no | |
748
+ | `review-id` | yes | no | |
749
+
750
+ ### Options
751
+
752
+ | Flag | Required | Default | Description |
753
+ | ------------------ | -------- | ------- | -------------------------------------------------------- |
754
+ | `--base-url <url>` | no | | Server base URL |
755
+ | `--json` | no | | Emit machine-readable JSON on stdout |
756
+ | `--status <csv>` | no | | Filter by item status (pending,approved,edited,rejected) |
757
+
758
+ ### `eigenpal workflow dataset review-request events [options] <automation-id> <review-id>`
759
+
760
+ List review activity, including example and field notes.
761
+
762
+ ### Arguments
763
+
764
+ | Name | Required | Variadic | Description |
765
+ | --------------- | -------- | -------- | ----------- |
766
+ | `automation-id` | yes | no | |
767
+ | `review-id` | yes | no | |
768
+
769
+ ### Options
770
+
771
+ | Flag | Required | Default | Description |
772
+ | ------------------ | -------- | ------- | ------------------------------------ |
773
+ | `--base-url <url>` | no | | Server base URL |
774
+ | `--json` | no | | Emit machine-readable JSON on stdout |
775
+
776
+ ### `eigenpal workflow dataset review-request item [options] <automation-id> <review-id> <item-id>`
777
+
778
+ Approve, reject, reopen, comment, edit, or record a field-decision on one review item.
779
+
780
+ ### Arguments
781
+
782
+ | Name | Required | Variadic | Description |
783
+ | --------------- | -------- | -------- | ----------- |
784
+ | `automation-id` | yes | no | |
785
+ | `review-id` | yes | no | |
786
+ | `item-id` | yes | no | |
787
+
788
+ ### Options
789
+
790
+ | Flag | Required | Default | Description |
791
+ | ------------------------------------------------------------------- | -------- | ------- | ------------------------------------------------------------------ |
792
+ | `--base-url <url>` | no | | Server base URL |
793
+ | `--json` | no | | Emit machine-readable JSON on stdout |
794
+ | `--action <approve\|reject\|reopen\|comment\|edit\|field-decision>` | yes | | Item action |
795
+ | `--expected-updated-at <iso>` | yes | | Item updatedAt the client last observed (optimistic concurrency) |
796
+ | `--comment <text>` | no | | Note stored on the item or field |
797
+ | `--field-path <path>` | no | | Dotted expected-output path for comment or field-decision |
798
+ | `--decision <approved\|rejected\|null>` | no | | Field decision for --action field-decision (null/clear removes it) |
799
+ | `--clear` | no | `false` | Clear a field decision (sends decision: null) |
800
+ | `--expected-json <json>` | no | | Replacement expected JSON when --action edit |
801
+
628
802
  ### `eigenpal workflow templates upload [options] <file>`
629
803
 
630
804
  Upload a DOCX or XLSX file as a new tmpl*… resource with an immutable tmpr*… revision.
package/dist/skill/reference/dataset-format.md CHANGED
@@ -300,6 +300,61 @@ For bulk changes, edit the local folder and re-push with
300
300
  still go through `dataset push`; CRUD only handles JSON input and
301
301
  `expected.json`-style outputs.
302
302
 
303
+ ## Dataset review requests
304
+
305
+ > **Review-first flow (agents: follow this automatically).** When building a
306
+ > new workflow or agent and you were given any examples: push the dataset,
307
+ > ship a V0, then request a dataset review as the first step — before
308
+ > iterating to perfection. Light iteration on an unreviewed dataset is fine,
309
+ > but evaluators, experiments, and versions must wait for reviewed ground
310
+ > truth: poll `.progress.complete`, reconcile manually, then close the request.
311
+
312
+ When reviewers need to approve, edit, or reject ground-truth expected
313
+ outputs without direct dataset write access, create a review request that
314
+ snapshots selected examples. The server copies each example's `input.json`
315
+ and `expected.json` values at creation time; file pointers stay as S3
316
+ references. Reviewers approve, reject (recommendation only — nothing is
317
+ deleted), leave comments, and record per-field decisions. There is **no
318
+ auto-apply** by design: after review, `dataset pull` and manually reconcile
319
+ each example into the live dataset (reviewers can err). Close via
320
+ `update --status closed`.
321
+
322
+ ```bash
323
+ # Ask a reviewer to inspect specific fields, with reasons, then poll until done.
324
+ eigenpal workflow dataset review-request create <automation-id> \
325
+ --title "Q1 invoice GT review" \
326
+ --example-name invoice-foo --example-name invoice-bar \
327
+ --instructions "Check IBAN checksums and invoice totals." \
328
+ --focus vendor.iban --focus-reason 'vendor.iban=OCR often mangles IBANs' \
329
+ --ignore currency \
330
+ --field-note 'invoice-foo.total=off by 0.01 last run' \
331
+ --status open --json
332
+
333
+ # List requests; `.progress.complete` means every example was decided.
334
+ eigenpal workflow dataset review-request list <automation-id> --status open --json
335
+
336
+ # Inspect items (includes fieldDecisions + inputDrifted), focus, and notes.
337
+ eigenpal workflow dataset review-request get <automation-id> <review-id> --json
338
+ eigenpal workflow dataset review-request events <automation-id> <review-id> --json
339
+
340
+ # Per-field decision (approved | rejected). --clear / --decision null removes it.
341
+ eigenpal workflow dataset review-request item <automation-id> <review-id> <item-id> \
342
+ --action field-decision --field-path vendor.iban --decision approved \
343
+ --expected-updated-at <iso> --json
344
+
345
+ # Per-example reject is a recommendation only — nothing is deleted.
346
+ eigenpal workflow dataset review-request item <automation-id> <review-id> <item-id> \
347
+ --action reject --comment "wrong vendor" --expected-updated-at <iso> --json
348
+
349
+ # Pull snapshots and MANUAL reconcile into the dataset, then close.
350
+ eigenpal workflow dataset pull <automation-id> --out ./dataset.zip
351
+ eigenpal workflow dataset review-request update <automation-id> <review-id> \
352
+ --status closed --json
353
+ ```
354
+
355
+ The same commands exist under `eigenpal agents dataset review-request …` for
356
+ agent automations.
357
+
303
358
  <!-- GENERATED:DATASET_REFERENCE START -->
304
359
  ## Schema reference
305
360
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@eigenpal/cli",
3
- "version": "0.19.1",
3
+ "version": "0.20.0",
4
4
  "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
5
5
  "keywords": [
6
6
  "eigenpal",