@eigenpal/cli 0.5.11 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/README.md +3 -1
- package/dist/cli.js +50602 -43044
- package/dist/skill/SKILL.md +152 -76
- package/dist/skill/reference/cli/agents.md +1330 -0
- package/dist/skill/reference/cli/auth.md +2 -2
- package/dist/skill/reference/cli/git.md +5 -3
- package/dist/skill/reference/cli/init.md +8 -2
- package/dist/skill/reference/cli/run.md +18 -0
- package/dist/skill/reference/cli/runs.md +23 -0
- package/dist/skill/reference/cli/skill.md +2 -2
- package/dist/skill/reference/cli/workflow.md +29 -29
- package/dist/skill/reference/dataset-format.md +63 -9
- package/dist/skill/reference/evaluators.md +8 -1
- package/dist/skill/reference/step-types.md +300 -18
- package/dist/skill/reference/workflow-yaml.md +2 -1
- package/package.json +1 -1
- package/dist/skill/reference/cli/agent.md +0 -1243
|
@@ -28,10 +28,10 @@ auth
|
|
|
28
28
|
### Core
|
|
29
29
|
|
|
30
30
|
| Command | Description |
|
|
31
|
-
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
31
|
+
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
32
32
|
| `eigenpal auth login [options]` | Authenticate with Eigenpal. Opens a browser to create an API key, stores it under a profile named after the tenant, and makes that profile active. Re-running `login` against the same tenant updates the existing profile in place. |
|
|
33
33
|
| `eigenpal auth logout [options] [profile]` | Remove a profile from the credentials file. Defaults to the active profile if no name is given. After removal the next available profile becomes active. |
|
|
34
|
-
| `eigenpal auth list
|
|
34
|
+
| `eigenpal auth list\|ls [options]` | List configured profiles. The active one is marked ● — switch with `auth use`. |
|
|
35
35
|
| `eigenpal auth use [options] [profile]` | Switch the active profile (persistent across shells). Omit `[profile]` to pick from a list. For one-shot per-shell switching, set `EIGENPAL_PROFILE=<name>` instead. |
|
|
36
36
|
|
|
37
37
|
## Details
|
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
# eigenpal git
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Passthrough to git with organization remote auth and committer identity.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
### Arguments
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
| Name | Required | Variadic | Description |
|
|
8
|
+
| --------- | -------- | -------- | ---------------------- |
|
|
9
|
+
| `gitArgs` | no | yes | Git arguments after -- |
|
|
@@ -26,7 +26,7 @@ init
|
|
|
26
26
|
| Command | Description |
|
|
27
27
|
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
28
28
|
| `eigenpal init workflow [options] <name>` | Alias of `eigenpal init <name>`. Kept so the `init {workflow,agent}` namespace stays visible — once the agent surface lights up, both kinds will live as siblings here. |
|
|
29
|
-
| `eigenpal init agent [options] <name>` |
|
|
29
|
+
| `eigenpal init agent [options] <name>` | Deprecated: use `eigenpal agents init <name> --template agent`. |
|
|
30
30
|
|
|
31
31
|
## Details
|
|
32
32
|
|
|
@@ -50,10 +50,16 @@ Alias of `eigenpal init <name>`. Kept so the `init {workflow,agent}` namespace s
|
|
|
50
50
|
|
|
51
51
|
### `eigenpal init agent [options] <name>`
|
|
52
52
|
|
|
53
|
-
|
|
53
|
+
Deprecated: use `eigenpal agents init <name> --template agent`.
|
|
54
54
|
|
|
55
55
|
### Arguments
|
|
56
56
|
|
|
57
57
|
| Name | Required | Variadic | Description |
|
|
58
58
|
| ------ | -------- | -------- | ----------- |
|
|
59
59
|
| `name` | yes | no | |
|
|
60
|
+
|
|
61
|
+
### Options
|
|
62
|
+
|
|
63
|
+
| Flag | Required | Default | Description |
|
|
64
|
+
| ------------- | -------- | ------- | ------------------------------------ |
|
|
65
|
+
| `--dir <dir>` | no | | Target directory (default: ./<name>) |
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# eigenpal run
|
|
2
|
+
|
|
3
|
+
Deprecated: use `eigenpal agents run <target>`.
|
|
4
|
+
|
|
5
|
+
### Arguments
|
|
6
|
+
|
|
7
|
+
| Name | Required | Variadic | Description |
|
|
8
|
+
| -------- | -------- | -------- | ----------- |
|
|
9
|
+
| `target` | yes | no | |
|
|
10
|
+
|
|
11
|
+
### Options
|
|
12
|
+
|
|
13
|
+
| Flag | Required | Default | Description |
|
|
14
|
+
| --------------------- | -------- | ------- | -------------------------------------------- |
|
|
15
|
+
| `--input-json <json>` | no | | JSON input object |
|
|
16
|
+
| `--input-file <path>` | no | | Input file to upload as multipart form-data |
|
|
17
|
+
| `--wait` | no | | Poll until the run reaches a terminal status |
|
|
18
|
+
| `--base-url <url>` | no | | Server base URL |
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# eigenpal runs
|
|
2
|
+
|
|
3
|
+
Deprecated: use `eigenpal agents runs list <target>`.
|
|
4
|
+
|
|
5
|
+
### Arguments
|
|
6
|
+
|
|
7
|
+
| Name | Required | Variadic | Description |
|
|
8
|
+
| -------- | -------- | -------- | ----------- |
|
|
9
|
+
| `target` | yes | no | |
|
|
10
|
+
|
|
11
|
+
### Options
|
|
12
|
+
|
|
13
|
+
| Flag | Required | Default | Description |
|
|
14
|
+
| --------------------- | -------- | ------- | ---------------------------- |
|
|
15
|
+
| `--status <status>` | no | | Filter by run status |
|
|
16
|
+
| `--include <items>` | no | | Comma-separated include list |
|
|
17
|
+
| `--compact` | no | | Render compact run rows |
|
|
18
|
+
| `--sort <field>` | no | | Sort field |
|
|
19
|
+
| `--order <asc\|desc>` | no | | Sort order |
|
|
20
|
+
| `--base-url <url>` | no | | Server base URL |
|
|
21
|
+
| `--json` | no | | Emit machine-readable JSON |
|
|
22
|
+
| `--limit <n>` | no | | Page size |
|
|
23
|
+
| `--offset <n>` | no | | Page offset |
|
|
@@ -26,10 +26,10 @@ skill
|
|
|
26
26
|
### Core
|
|
27
27
|
|
|
28
28
|
| Command | Description |
|
|
29
|
-
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
29
|
+
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
30
30
|
| `eigenpal skill install [options]` | Install the skill into one or more tools. Opens an interactive multiselect picker (toggle on to install, toggle off to uninstall) — pass `--tools` for non-interactive use. |
|
|
31
31
|
| `eigenpal skill uninstall [options] [toolIds...]` | Remove the skill from named tools (e.g. `skill uninstall claude cursor`), pass `--all` to wipe every install, or run with no args in a TTY for an interactive picker showing only installed tools. |
|
|
32
|
-
| `eigenpal skill list
|
|
32
|
+
| `eigenpal skill list\|ls [options]` | Show every tool that has an Eigenpal skill installed in the current directory, with the CLI version that wrote it. |
|
|
33
33
|
|
|
34
34
|
## Details
|
|
35
35
|
|
|
@@ -111,8 +111,8 @@ workflow
|
|
|
111
111
|
### Core
|
|
112
112
|
|
|
113
113
|
| Command | Description |
|
|
114
|
-
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
115
|
-
| `eigenpal workflow list
|
|
114
|
+
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
115
|
+
| `eigenpal workflow list\|ls [options]` | List workflows the caller can read. |
|
|
116
116
|
| `eigenpal workflow pull [options] <workflow-id>` | Download the YAML definition of the workflow at its current version. |
|
|
117
117
|
| `eigenpal workflow push [options]` | Create or update a workflow from a YAML file. |
|
|
118
118
|
| `eigenpal workflow move [options] <workflow-id>` | Move a workflow to a folder path, creating folders as needed |
|
|
@@ -130,8 +130,8 @@ workflow
|
|
|
130
130
|
### Dataset
|
|
131
131
|
|
|
132
132
|
| Command | Description |
|
|
133
|
-
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
|
|
134
|
-
| `eigenpal workflow dataset list
|
|
133
|
+
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
|
|
134
|
+
| `eigenpal workflow dataset list\|ls [options] <workflow-id>` | List eval examples for the workflow. |
|
|
135
135
|
| `eigenpal workflow dataset pull [options] <workflow-id>` | Download the workflow's dataset as a ZIP archive. |
|
|
136
136
|
| `eigenpal workflow dataset push [options] <workflow-id>` | Replace or extend the workflow's dataset from a ZIP or folder. |
|
|
137
137
|
| `eigenpal workflow dataset example create [options] <workflow-id>` | Create one eval example without re-uploading the dataset. |
|
|
@@ -142,46 +142,46 @@ workflow
|
|
|
142
142
|
|
|
143
143
|
### Experiment
|
|
144
144
|
|
|
145
|
-
| Command
|
|
146
|
-
|
|
|
147
|
-
| `eigenpal workflow experiment
|
|
148
|
-
| `eigenpal workflow experiment
|
|
149
|
-
| `eigenpal workflow experiment
|
|
150
|
-
| `eigenpal workflow experiment
|
|
151
|
-
| `eigenpal workflow experiment
|
|
152
|
-
| `eigenpal workflow experiment
|
|
153
|
-
| `eigenpal workflow experiment
|
|
145
|
+
| Command | Description |
|
|
146
|
+
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
147
|
+
| `eigenpal workflow experiment\|exp list\|ls [options] <workflow-id>` | List executions for the workflow, newest first. |
|
|
148
|
+
| `eigenpal workflow experiment\|exp run [options] <workflow-id>` | Start a batch eval against the workflow's dataset. |
|
|
149
|
+
| `eigenpal workflow experiment\|exp status [options] <workflow-id> <batchId>` | Aggregate progress for a batch by `batchId`. |
|
|
150
|
+
| `eigenpal workflow experiment\|exp cancel [options] <workflow-id> <batchId>` | Cancel every execution in a batch. Idempotent. |
|
|
151
|
+
| `eigenpal workflow experiment\|exp results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON. |
|
|
152
|
+
| `eigenpal workflow experiment\|exp compare\|diff [options] <batchIdA> <batchIdB>` | Diff eval scores between two experiment batches. |
|
|
153
|
+
| `eigenpal workflow experiment\|exp watch [options] <workflow-id> <batchId>` | Poll until terminal, then auto-pull results — replaces `status --watch` + `results --out`. |
|
|
154
154
|
|
|
155
155
|
### Execution
|
|
156
156
|
|
|
157
|
-
| Command
|
|
158
|
-
|
|
|
159
|
-
| `eigenpal workflow execution
|
|
160
|
-
| `eigenpal workflow execution
|
|
161
|
-
| `eigenpal workflow execution
|
|
162
|
-
| `eigenpal workflow execution
|
|
163
|
-
| `eigenpal workflow execution
|
|
164
|
-
| `eigenpal workflow execution
|
|
157
|
+
| Command | Description |
|
|
158
|
+
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
|
|
159
|
+
| `eigenpal workflow execution\|exec run [options] <workflow-id> [examples...]` | Run a saved workflow against local dataset examples. |
|
|
160
|
+
| `eigenpal workflow execution\|exec get [options] <executionId>` | Fetch a single execution payload. Optionally narrow to one step. |
|
|
161
|
+
| `eigenpal workflow execution\|exec list\|ls [options] <workflow-id>` | List recent executions for a workflow. |
|
|
162
|
+
| `eigenpal workflow execution\|exec watch [options] <executionId>` | Stream live execution status until terminal or 30-min detach. |
|
|
163
|
+
| `eigenpal workflow execution\|exec compare\|diff [options] <executionA> <executionB>` | Diff two executions side-by-side, per step. |
|
|
164
|
+
| `eigenpal workflow execution\|exec cancel [options] <executionId>` | Cancel an execution. Idempotent on already-terminal runs. |
|
|
165
165
|
|
|
166
166
|
### Versions
|
|
167
167
|
|
|
168
|
-
| Command | Description
|
|
169
|
-
| ------------------------------------------------------------------------ |
|
|
170
|
-
| `eigenpal workflow versions list
|
|
171
|
-
| `eigenpal workflow versions restore [options] <workflow-id> <versionId>` | Restore the workflow to a previous version.
|
|
168
|
+
| Command | Description |
|
|
169
|
+
| ------------------------------------------------------------------------ | ------------------------------------------------ |
|
|
170
|
+
| `eigenpal workflow versions list\|ls [options] <workflow-id>` | List historical workflow versions, newest first. |
|
|
171
|
+
| `eigenpal workflow versions restore [options] <workflow-id> <versionId>` | Restore the workflow to a previous version. |
|
|
172
172
|
|
|
173
173
|
### Step-type
|
|
174
174
|
|
|
175
175
|
| Command | Description |
|
|
176
|
-
| -------------------------------------------------- | ------------------------------------------------------------- |
|
|
177
|
-
| `eigenpal workflow step-type list
|
|
176
|
+
| -------------------------------------------------- | ------------------------------------------------------------- |
|
|
177
|
+
| `eigenpal workflow step-type list\|ls [options]` | List every step type the deployment supports. |
|
|
178
178
|
| `eigenpal workflow step-type get [options] <type>` | Return the full schema and behavioral docs for one step type. |
|
|
179
179
|
|
|
180
180
|
### Evaluator-type
|
|
181
181
|
|
|
182
182
|
| Command | Description |
|
|
183
|
-
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
|
|
184
|
-
| `eigenpal workflow evaluator-type list
|
|
183
|
+
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
|
|
184
|
+
| `eigenpal workflow evaluator-type list\|ls [options]` | List every evaluator type with a one-line description. |
|
|
185
185
|
| `eigenpal workflow evaluator-type get [options] <type>` | Fetch the JSON Schema for one evaluator type. Pipe through `jq` to inspect specific fields. |
|
|
186
186
|
|
|
187
187
|
### Step
|
|
@@ -27,19 +27,28 @@ dataset/
|
|
|
27
27
|
│ │ ├── Contract_2026.pdf
|
|
28
28
|
│ │ └── Appendix.pdf
|
|
29
29
|
│ ├── expected/
|
|
30
|
-
│ │ ├── output.json OPTIONAL — raw ground truth JSON
|
|
30
|
+
│ │ ├── output.json OPTIONAL — raw ground truth JSON (success-expected)
|
|
31
31
|
│ │ └── generated_invoice/ OPTIONAL — expected file output
|
|
32
32
|
│ │ └── Invoice.docx
|
|
33
33
|
│ └── meta.json OPTIONAL — { rowOrder?, annotation?, overrides? }
|
|
34
34
|
│
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
35
|
+
├── invoice-bar/
|
|
36
|
+
│ └── input/
|
|
37
|
+
│ └── arguments.json { amount: 42 }
|
|
38
|
+
│
|
|
39
|
+
└── unsupported-format/
|
|
40
|
+
├── input/
|
|
41
|
+
│ └── arguments.json
|
|
42
|
+
└── expected/
|
|
43
|
+
└── error.json OPTIONAL — failure-expected, mutually exclusive with output.json
|
|
38
44
|
```
|
|
39
45
|
|
|
40
|
-
The `expected/` subfolder mirrors `input/` exactly
|
|
46
|
+
The `expected/` subfolder mirrors `input/` exactly. `output.json` is
|
|
41
47
|
the scalar/object data ground truth, and any expected file outputs go
|
|
42
48
|
in `expected/<docKey>/<file>` folders (one folder per output path).
|
|
49
|
+
`error.json` is the failure-expected counterpart (see "Failure-expected
|
|
50
|
+
examples" below) and cannot coexist with `output.json` for the same
|
|
51
|
+
example.
|
|
43
52
|
|
|
44
53
|
## Rules (every one is enforced server-side at import)
|
|
45
54
|
|
|
@@ -66,9 +75,29 @@ in `expected/<docKey>/<file>` folders (one folder per output path).
|
|
|
66
75
|
`{"contract": "..."}` AND `input/contract/file.pdf` exists, import
|
|
67
76
|
fails with `code: argument_name_collision`.
|
|
68
77
|
- **`expected/output.json`** is OPTIONAL. When present, it must be a
|
|
69
|
-
JSON object that mirrors your workflow's `output:` shape 1:1
|
|
78
|
+
JSON object that mirrors your workflow's `output:` shape 1:1, no
|
|
70
79
|
envelope, no wrapper. A scalar/array at the top level is rejected
|
|
71
|
-
with `code: invalid_expected_output`.
|
|
80
|
+
with `code: invalid_expected_output`. Mutually exclusive with
|
|
81
|
+
`expected/error.json` (see below).
|
|
82
|
+
- **`expected/error.json`** is OPTIONAL and marks the example as
|
|
83
|
+
failure-expected (the workflow is supposed to terminate via
|
|
84
|
+
`control.fail`). Must be a JSON object with at least one of
|
|
85
|
+
`{ code, messageContains, step }`; an empty object is rejected (it
|
|
86
|
+
would match any failure). When set, the scorer matches the example's
|
|
87
|
+
terminal `executions.error` envelope against the fields you specify
|
|
88
|
+
and ignores `expected/output.json` entirely. `code` is the HTTP-style
|
|
89
|
+
status code from `control.fail`'s `statusCode`; `messageContains` is a
|
|
90
|
+
case-sensitive substring of the failure message; `step` is the name
|
|
91
|
+
of the `control.fail` step expected to fire.
|
|
92
|
+
|
|
93
|
+
```json
|
|
94
|
+
// expected/error.json
|
|
95
|
+
{
|
|
96
|
+
"code": 422,
|
|
97
|
+
"messageContains": "unsupported document type",
|
|
98
|
+
"step": "reject-unsupported"
|
|
99
|
+
}
|
|
100
|
+
```
|
|
72
101
|
|
|
73
102
|
> **Note:** the importer is idempotent and accepts both bare objects and
|
|
74
103
|
> pre-wrapped `{ data: { ... } }` forms; bare is preferred.
|
|
@@ -145,9 +174,34 @@ The server materializes:
|
|
|
145
174
|
|
|
146
175
|
Each file is uploaded to S3 with a fresh `fileId`; the original
|
|
147
176
|
filename is preserved on the `files` row and shows up in the dashboard.
|
|
148
|
-
The `expectedOutput` envelope (`{ data?, expectedDocuments? }`) is
|
|
177
|
+
The `expectedOutput` envelope (`{ data?, expectedDocuments?, error? }`) is
|
|
149
178
|
internal storage shape — your dataset folder stays clean (raw
|
|
150
|
-
`output.json`
|
|
179
|
+
`output.json` or `error.json`, plus per-docKey folders).
|
|
180
|
+
|
|
181
|
+
## Failure-expected examples
|
|
182
|
+
|
|
183
|
+
When a workflow uses `control.fail` to reject bad inputs, you usually
|
|
184
|
+
want to test both paths: happy-path examples carry `expected/output.json`
|
|
185
|
+
and assert the workflow completed; rejection examples carry
|
|
186
|
+
`expected/error.json` and assert the workflow failed with the right
|
|
187
|
+
status code.
|
|
188
|
+
|
|
189
|
+
```
|
|
190
|
+
examples/
|
|
191
|
+
├── valid-invoice/
|
|
192
|
+
│ ├── input/...
|
|
193
|
+
│ └── expected/output.json # success-expected
|
|
194
|
+
└── unsupported-format/
|
|
195
|
+
├── input/...
|
|
196
|
+
└── expected/error.json # failure-expected
|
|
197
|
+
# { "code": 422, "step": "reject-unsupported" }
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
The two files are mutually exclusive for the same example (an example
|
|
201
|
+
is either a success-case or a failure-case, never both). The `exact-diff`
|
|
202
|
+
evaluator branches on the envelope shape and reports a clean
|
|
203
|
+
`expectedFailureButSucceeded` or `actualWasUntypedError` reason when
|
|
204
|
+
the actual run doesn't match the expectation.
|
|
151
205
|
|
|
152
206
|
## Validate before pushing
|
|
153
207
|
|
|
@@ -54,7 +54,7 @@ finance) who never see the YAML or the judge prompt. Write it for them:
|
|
|
54
54
|
|
|
55
55
|
## Evaluator types
|
|
56
56
|
|
|
57
|
-
### `exact-diff` — JSON deep-diff against `expected/output.json`
|
|
57
|
+
### `exact-diff` — JSON deep-diff against `expected/output.json` or `expected/error.json`
|
|
58
58
|
|
|
59
59
|
```yaml
|
|
60
60
|
- name: invoice-fields
|
|
@@ -70,6 +70,13 @@ finance) who never see the YAML or the judge prompt. Write it for them:
|
|
|
70
70
|
When set, only those paths are diffed. When absent, the whole object
|
|
71
71
|
is compared.
|
|
72
72
|
|
|
73
|
+
`exact-diff` branches on the example's expected shape: success-expected
|
|
74
|
+
examples (`expected/output.json`) are diffed against the workflow's actual
|
|
75
|
+
output, and failure-expected examples (`expected/error.json`, see
|
|
76
|
+
`reference/dataset-format.md`) are matched against the `control.fail`
|
|
77
|
+
envelope persisted to `executions.error`. The same evaluator handles both
|
|
78
|
+
paths; you do not need a separate evaluator for failure cases.
|
|
79
|
+
|
|
73
80
|
### `llm-judge` — LLM-as-judge scoring
|
|
74
81
|
|
|
75
82
|
```yaml
|