@eigenpal/cli 0.7.2 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +89 -23
- package/README.md +8 -8
- package/dist/cli.js +9024 -8865
- package/dist/skill/SKILL.md +87 -415
- package/dist/skill/reference/cli/agents.md +25 -25
- package/dist/skill/reference/cli/runs.md +21 -0
- package/dist/skill/reference/cli/surface.md +1 -2
- package/dist/skill/reference/cli/workflow.md +10 -10
- package/dist/skill/reference/dataset-format.md +6 -6
- package/dist/skill/reference/debugging.md +5 -5
- package/dist/skill/reference/evaluators.md +7 -7
- package/dist/skill/reference/step-exec.md +2 -2
- package/dist/skill/reference/step-types.md +5 -5
- package/dist/skill/reference/workflow-yaml.md +4 -4
- package/dist/templates/blank/README.md +1 -1
- package/package.json +1 -1
- package/dist/skill/reference/cli/completion.md +0 -9
- package/dist/skill/reference/cli/git.md +0 -9
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# eigenpal agents
|
|
2
2
|
|
|
3
|
-
Manage Eigenpal agents: Git source, datasets,
|
|
3
|
+
Manage Eigenpal agents: Git source, datasets, experiments, sessions, and releases.
|
|
4
4
|
|
|
5
5
|
## Contents
|
|
6
6
|
|
|
@@ -157,20 +157,20 @@ agents
|
|
|
157
157
|
| Command | Description |
|
|
158
158
|
| --------------------------------------------------------------- | -------------------------------------------------------------------------- |
|
|
159
159
|
| `eigenpal agents dataset list\|ls [options] <agent-id-or-slug>` | List dataset examples for an agent. |
|
|
160
|
-
| `eigenpal agents dataset push [options] <agent-id-or-slug>` | Upload dataset examples from a local dataset directory.
|
|
161
|
-
| `eigenpal agents dataset pull [options] <agent-id-or-slug>` | Download an agent dataset
|
|
160
|
+
| `eigenpal agents dataset push [options] <agent-id-or-slug>` | Upload dataset examples from a local dataset directory or zip archive. |
|
|
161
|
+
| `eigenpal agents dataset pull [options] <agent-id-or-slug>` | Download an agent dataset as a .zip archive. |
|
|
162
162
|
| `eigenpal agents dataset validate [options] [path]` | Validate a local dataset directory against the agent input/output schemas. |
|
|
163
163
|
|
|
164
164
|
### Experiment
|
|
165
165
|
|
|
166
|
-
| Command | Description
|
|
167
|
-
| ----------------------------------------------------------------------------------- |
|
|
168
|
-
| `eigenpal agents experiment\|exp run [options] <agent-id-or-slug>` | Start an experiment over dataset examples.
|
|
169
|
-
| `eigenpal agents experiment\|exp status [options] <agent-id-or-slug> <batch-id>` | Get experiment status.
|
|
170
|
-
| `eigenpal agents experiment\|exp results [options] <agent-id-or-slug> [batch-id]` | Print experiment results as JSON or CSV.
|
|
171
|
-
| `eigenpal agents experiment\|exp list\|ls [options] <agent-id-or-slug>` | List experiments.
|
|
172
|
-
| `eigenpal agents experiment\|exp compare\|diff [options] <batch-id-a> <batch-id-b>` |
|
|
173
|
-
| `eigenpal agents experiment\|exp cancel [options] <agent-id-or-slug> <batch-id>` | Cancel every active execution in an experiment.
|
|
166
|
+
| Command | Description |
|
|
167
|
+
| ----------------------------------------------------------------------------------- | ------------------------------------------------ |
|
|
168
|
+
| `eigenpal agents experiment\|exp run [options] <agent-id-or-slug>` | Start an experiment over dataset examples. |
|
|
169
|
+
| `eigenpal agents experiment\|exp status [options] <agent-id-or-slug> <batch-id>` | Get experiment status. |
|
|
170
|
+
| `eigenpal agents experiment\|exp results [options] <agent-id-or-slug> [batch-id]` | Print experiment results as JSON or CSV. |
|
|
171
|
+
| `eigenpal agents experiment\|exp list\|ls [options] <agent-id-or-slug>` | List experiments. |
|
|
172
|
+
| `eigenpal agents experiment\|exp compare\|diff [options] <batch-id-a> <batch-id-b>` | Diff eval scores between two experiment batches. |
|
|
173
|
+
| `eigenpal agents experiment\|exp cancel [options] <agent-id-or-slug> <batch-id>` | Cancel every active execution in an experiment. |
|
|
174
174
|
|
|
175
175
|
### Session
|
|
176
176
|
|
|
@@ -572,7 +572,7 @@ List dataset examples for an agent.
|
|
|
572
572
|
|
|
573
573
|
### `eigenpal agents dataset push [options] <agent-id-or-slug>`
|
|
574
574
|
|
|
575
|
-
Upload dataset examples from a local dataset directory.
|
|
575
|
+
Upload dataset examples from a local dataset directory or zip archive.
|
|
576
576
|
|
|
577
577
|
### Arguments
|
|
578
578
|
|
|
@@ -586,13 +586,13 @@ Upload dataset examples from a local dataset directory.
|
|
|
586
586
|
| -------------------------- | -------- | ---------- | ---------------------------------------------------- |
|
|
587
587
|
| `--base-url <url>` | no | | Server base URL |
|
|
588
588
|
| `--json` | no | | Output the raw server response as JSON |
|
|
589
|
-
| `--file <path>` | yes | | Dataset directory
|
|
589
|
+
| `--file <path>` | yes | | Dataset directory or .zip archive |
|
|
590
590
|
| `--mode <append\|replace>` | no | `"append"` | Upload mode |
|
|
591
591
|
| `--yes` | no | | Confirm replace mode in non-interactive environments |
|
|
592
592
|
|
|
593
593
|
### `eigenpal agents dataset pull [options] <agent-id-or-slug>`
|
|
594
594
|
|
|
595
|
-
Download an agent dataset
|
|
595
|
+
Download an agent dataset as a .zip archive.
|
|
596
596
|
|
|
597
597
|
### Arguments
|
|
598
598
|
|
|
@@ -602,10 +602,10 @@ Download an agent dataset directory.
|
|
|
602
602
|
|
|
603
603
|
### Options
|
|
604
604
|
|
|
605
|
-
| Flag | Required | Default | Description
|
|
606
|
-
| ------------------ | -------- | ----------- |
|
|
607
|
-
| `--base-url <url>` | no | | Server base URL
|
|
608
|
-
| `--out <
|
|
605
|
+
| Flag | Required | Default | Description |
|
|
606
|
+
| ------------------ | -------- | ----------- | --------------------------------------- |
|
|
607
|
+
| `--base-url <url>` | no | | Server base URL |
|
|
608
|
+
| `--out <path>` | no | `"dataset"` | Output .zip path (default: dataset.zip) |
|
|
609
609
|
|
|
610
610
|
### `eigenpal agents dataset validate [options] [path]`
|
|
611
611
|
|
|
@@ -707,7 +707,7 @@ List experiments.
|
|
|
707
707
|
|
|
708
708
|
### `eigenpal agents experiment|exp compare|diff [options] <batch-id-a> <batch-id-b>`
|
|
709
709
|
|
|
710
|
-
|
|
710
|
+
Diff eval scores between two experiment batches.
|
|
711
711
|
|
|
712
712
|
### Arguments
|
|
713
713
|
|
|
@@ -718,12 +718,12 @@ Compare two experiment batches.
|
|
|
718
718
|
|
|
719
719
|
### Options
|
|
720
720
|
|
|
721
|
-
| Flag
|
|
722
|
-
|
|
|
723
|
-
| `--base-url <url>`
|
|
724
|
-
| `--json`
|
|
725
|
-
| `--sort <
|
|
726
|
-
| `--regression-threshold <n>`
|
|
721
|
+
| Flag | Required | Default | Description |
|
|
722
|
+
| ------------------------------------------------------ | -------- | ------------------ | ------------------------------------------------------ |
|
|
723
|
+
| `--base-url <url>` | no | | Server base URL |
|
|
724
|
+
| `--json` | no | | Output the raw server response as JSON |
|
|
725
|
+
| `--sort <abs-delta-desc\|delta-asc\|delta-desc\|name>` | no | `"abs-delta-desc"` | Row sort order (default: biggest movers first) |
|
|
726
|
+
| `--regression-threshold <n>` | no | `0.05` | Δ below this is flagged as a regression (default 0.05) |
|
|
727
727
|
|
|
728
728
|
### `eigenpal agents experiment|exp cancel [options] <agent-id-or-slug> <batch-id>`
|
|
729
729
|
|
|
@@ -15,6 +15,7 @@ Inspect, watch, and manage workflow, agent, and eval runs.
|
|
|
15
15
|
- [`eigenpal runs get [options] <run-id>`](#eigenpal-runs-get-options-run-id)
|
|
16
16
|
- [`eigenpal runs compare|diff [options] <reference-run-id> <run-id>`](#eigenpal-runs-comparediff-options-reference-run-id-run-id)
|
|
17
17
|
- [`eigenpal runs rerun [options] <run-id>`](#eigenpal-runs-rerun-options-run-id)
|
|
18
|
+
- [`eigenpal runs promote [options] <run-id>`](#eigenpal-runs-promote-options-run-id)
|
|
18
19
|
- [`eigenpal runs trace [options] <run-id>`](#eigenpal-runs-trace-options-run-id)
|
|
19
20
|
- [`eigenpal runs watch [options] <run-id>`](#eigenpal-runs-watch-options-run-id)
|
|
20
21
|
- [`eigenpal runs cancel [options] <run-id>`](#eigenpal-runs-cancel-options-run-id)
|
|
@@ -38,6 +39,7 @@ runs
|
|
|
38
39
|
├── get <run-id>
|
|
39
40
|
├── compare|diff <reference-run-id> <run-id>
|
|
40
41
|
├── rerun <run-id>
|
|
42
|
+
├── promote <run-id>
|
|
41
43
|
├── artifacts|artifact
|
|
42
44
|
│ ├── list|ls <run-id>
|
|
43
45
|
│ └── fetch <run-id>
|
|
@@ -67,6 +69,7 @@ runs
|
|
|
67
69
|
| `eigenpal runs get [options] <run-id>` | Get one run. |
|
|
68
70
|
| `eigenpal runs compare\|diff [options] <reference-run-id> <run-id>` | Compare one run against another run. PDF/DOCX text comparison uses pdftotext/python3 and reports byte fallbacks. |
|
|
69
71
|
| `eigenpal runs rerun [options] <run-id>` | Create a new run from a previous run's stored input snapshot. |
|
|
72
|
+
| `eigenpal runs promote [options] <run-id>` | Promote a run into a dataset example on the run's automation. |
|
|
70
73
|
| `eigenpal runs trace [options] <run-id>` | Print raw trace.jsonl for a run, or write it with --out. |
|
|
71
74
|
| `eigenpal runs watch [options] <run-id>` | Watch a run until it reaches a terminal status. |
|
|
72
75
|
| `eigenpal runs cancel [options] <run-id>` | Cancel a run. |
|
|
@@ -198,6 +201,24 @@ Create a new run from a previous run's stored input snapshot.
|
|
|
198
201
|
| `--interval <seconds>` | no | `2` | Polling interval in seconds |
|
|
199
202
|
| `--max-wait <seconds>` | no | `1800` | Maximum wait before exiting 2 |
|
|
200
203
|
|
|
204
|
+
### `eigenpal runs promote [options] <run-id>`
|
|
205
|
+
|
|
206
|
+
Promote a run into a dataset example on the run's automation.
|
|
207
|
+
|
|
208
|
+
### Arguments
|
|
209
|
+
|
|
210
|
+
| Name | Required | Variadic | Description |
|
|
211
|
+
| -------- | -------- | -------- | ----------- |
|
|
212
|
+
| `run-id` | yes | no | |
|
|
213
|
+
|
|
214
|
+
### Options
|
|
215
|
+
|
|
216
|
+
| Flag | Required | Default | Description |
|
|
217
|
+
| ------------------ | -------- | ------- | -------------------------------------- |
|
|
218
|
+
| `--base-url <url>` | no | | Server base URL |
|
|
219
|
+
| `--json` | no | | Output the raw server response as JSON |
|
|
220
|
+
| `--name <name>` | no | | Name for the created example |
|
|
221
|
+
|
|
201
222
|
### `eigenpal runs trace [options] <run-id>`
|
|
202
223
|
|
|
203
224
|
Print raw trace.jsonl for a run, or write it with --out.
|
|
@@ -108,6 +108,7 @@ eigenpal
|
|
|
108
108
|
│ ├── get <run-id>
|
|
109
109
|
│ ├── compare|diff <reference-run-id> <run-id>
|
|
110
110
|
│ ├── rerun <run-id>
|
|
111
|
+
│ ├── promote <run-id>
|
|
111
112
|
│ ├── artifacts|artifact
|
|
112
113
|
│ │ ├── list|ls <run-id>
|
|
113
114
|
│ │ └── fetch <run-id>
|
|
@@ -125,8 +126,6 @@ eigenpal
|
|
|
125
126
|
│ │ └── delete <run-id> <name>
|
|
126
127
|
│ ├── watch <run-id>
|
|
127
128
|
│ └── cancel <run-id>
|
|
128
|
-
├── git [gitArgs...]
|
|
129
|
-
├── completion <shell>
|
|
130
129
|
└── skill
|
|
131
130
|
├── install
|
|
132
131
|
├── uninstall [toolIds...]
|
|
@@ -96,14 +96,14 @@ workflow
|
|
|
96
96
|
|
|
97
97
|
### Core
|
|
98
98
|
|
|
99
|
-
| Command | Description
|
|
100
|
-
| ------------------------------------------------------- |
|
|
101
|
-
| `eigenpal workflow list\|ls [options]` | List workflows the caller can read.
|
|
102
|
-
| `eigenpal workflow pull [options] <workflow-id>` | Download the YAML definition of the workflow at its current version.
|
|
103
|
-
| `eigenpal workflow push [options]` | Create or update a workflow from a YAML file.
|
|
104
|
-
| `eigenpal workflow move [options] <workflow-id>` | Move a workflow to a folder path, creating folders as needed
|
|
105
|
-
| `eigenpal workflow validate [options] [path]` | Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/). With [path] pointing at a YAML file: validates
|
|
106
|
-
| `eigenpal workflow clear-local [options] [examples...]` | Delete local execution artifacts under ./dataset/examples/. Keeps the latest run per example by default.
|
|
99
|
+
| Command | Description |
|
|
100
|
+
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
101
|
+
| `eigenpal workflow list\|ls [options]` | List workflows the caller can read. |
|
|
102
|
+
| `eigenpal workflow pull [options] <workflow-id>` | Download the YAML definition of the workflow at its current version. |
|
|
103
|
+
| `eigenpal workflow push [options]` | Create or update a workflow from a YAML file. |
|
|
104
|
+
| `eigenpal workflow move [options] <workflow-id>` | Move a workflow to a folder path, creating folders as needed |
|
|
105
|
+
| `eigenpal workflow validate [options] [path]` | Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/). With [path] pointing at a YAML file: validates that workflow.yaml. For per-noun targeting use `evaluators validate` or `dataset validate`. |
|
|
106
|
+
| `eigenpal workflow clear-local [options] [examples...]` | Delete local execution artifacts under ./dataset/examples/. Keeps the latest run per example by default. |
|
|
107
107
|
|
|
108
108
|
### Evaluators
|
|
109
109
|
|
|
@@ -233,7 +233,7 @@ Move a workflow to a folder path, creating folders as needed
|
|
|
233
233
|
|
|
234
234
|
### `eigenpal workflow validate [options] [path]`
|
|
235
235
|
|
|
236
|
-
Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/). With [path] pointing at a YAML file: validates
|
|
236
|
+
Local-only validation. Without [path]: runs the templated three-way check (./workflow.yaml + ./evaluators.yaml + ./dataset/). With [path] pointing at a YAML file: validates that workflow.yaml. For per-noun targeting use `evaluators validate` or `dataset validate`.
|
|
237
237
|
|
|
238
238
|
### Arguments
|
|
239
239
|
|
|
@@ -601,7 +601,7 @@ Poll until terminal, then auto-pull results — replaces `status --watch` + `res
|
|
|
601
601
|
| `--max-wait <seconds>` | no | `1800` | Hard ceiling in seconds (default 1800 = 30 min) |
|
|
602
602
|
| `--pull-on-complete <path>` | no | | Destination for the results file. Default: ./results-<batchId>.<format> |
|
|
603
603
|
| `--format <csv\|json>` | no | `"json"` | Results export format (default json) |
|
|
604
|
-
| `--no-pull` | no | | Skip auto-pulling results on terminal (
|
|
604
|
+
| `--no-pull` | no | | Skip auto-pulling results on terminal (watch only) |
|
|
605
605
|
| `--base-url <url>` | no | | Server base URL |
|
|
606
606
|
|
|
607
607
|
### `eigenpal workflow versions list|ls [options] <workflow-id>`
|
|
@@ -54,8 +54,8 @@ example.
|
|
|
54
54
|
|
|
55
55
|
- **Folder names** (example names, argument names, expected document
|
|
56
56
|
keys) must match `[a-z0-9][a-z0-9-_]*`.
|
|
57
|
-
- **`input/arguments.json`** is REQUIRED for every example, even if it
|
|
58
|
-
|
|
57
|
+
- **`input/arguments.json`** is REQUIRED for every example, even if it is
|
|
58
|
+
`{}`. It must be a JSON object (not array, not literal).
|
|
59
59
|
- **File-arguments** live in `input/<arg-name>/<filename>`. The
|
|
60
60
|
materialized shape on `triggerInput` is **driven by the workflow YAML's
|
|
61
61
|
`inputs[].type`**, not by the file count:
|
|
@@ -201,7 +201,7 @@ The two files are mutually exclusive for the same example (an example
|
|
|
201
201
|
is either a success-case or a failure-case, never both). The `exact-diff`
|
|
202
202
|
evaluator branches on the envelope shape and reports a clean
|
|
203
203
|
`expectedFailureButSucceeded` or `actualWasUntypedError` reason when
|
|
204
|
-
the actual run
|
|
204
|
+
the actual run does not match the expectation.
|
|
205
205
|
|
|
206
206
|
## Validate before pushing
|
|
207
207
|
|
|
@@ -238,7 +238,7 @@ includes `{ created, expectedSet, … }`:
|
|
|
238
238
|
|
|
239
239
|
## Editing a dataset on the server
|
|
240
240
|
|
|
241
|
-
To inspect or round-trip what
|
|
241
|
+
To inspect or round-trip what is currently on the server:
|
|
242
242
|
|
|
243
243
|
```bash
|
|
244
244
|
eigenpal workflow dataset list <workflow-id>
|
|
@@ -278,7 +278,7 @@ eigenpal workflow dataset example update <workflow-id> <example-id> \
|
|
|
278
278
|
--expected-file /tmp/correct.json
|
|
279
279
|
```
|
|
280
280
|
|
|
281
|
-
`--expected-file -` reads JSON from stdin if you
|
|
281
|
+
`--expected-file -` reads JSON from stdin if you would rather pipe. Pass
|
|
282
282
|
empty string to clear an annotation: `--annotation ""`.
|
|
283
283
|
|
|
284
284
|
### Add one new example
|
|
@@ -292,7 +292,7 @@ eigenpal workflow dataset example create <workflow-id> \
|
|
|
292
292
|
```
|
|
293
293
|
|
|
294
294
|
`--input-json` / `--expected-json` accept inline JSON literals; the
|
|
295
|
-
`-file` variants read from disk (or `-` for stdin). They
|
|
295
|
+
`-file` variants read from disk (or `-` for stdin). They are mutually
|
|
296
296
|
exclusive — pass one or the other, not both.
|
|
297
297
|
|
|
298
298
|
File-arg uploads still go through `dataset push`; CRUD only handles
|
|
@@ -54,7 +54,7 @@ The grouped run payload includes:
|
|
|
54
54
|
substitution. Lets you see what the LLM actually got, what URL
|
|
55
55
|
HTTP actually hit, what items the script actually saw. Populated
|
|
56
56
|
for `ai.*` / `transform.*` / `action.*`. Null for `control.*`
|
|
57
|
-
(orchestration steps
|
|
57
|
+
(orchestration steps do not have a meaningful resolved config).
|
|
58
58
|
- `output` — what the step returned
|
|
59
59
|
- `error` — error message + code when `status: 'failed'`
|
|
60
60
|
- `durationMs`
|
|
@@ -108,7 +108,7 @@ no network or filesystem, so most timeouts are infinite-loop bugs.
|
|
|
108
108
|
|
|
109
109
|
### `step_type_unknown`
|
|
110
110
|
|
|
111
|
-
`workflow.yaml` references a `type:` that the deployment
|
|
111
|
+
`workflow.yaml` references a `type:` that the deployment does not know
|
|
112
112
|
about. Common after a typo or a step type that was renamed.
|
|
113
113
|
|
|
114
114
|
```bash
|
|
@@ -219,7 +219,7 @@ broken. If you see:
|
|
|
219
219
|
Set EIGENPAL_BASE_URL or pass --base-url <url>.
|
|
220
220
|
```
|
|
221
221
|
|
|
222
|
-
That
|
|
222
|
+
That is a **connection** error — the URL is wrong, the server is down,
|
|
223
223
|
or your network is blocked. The URL the CLI tried is echoed back so a
|
|
224
224
|
stale `EIGENPAL_BASE_URL` is immediately obvious.
|
|
225
225
|
|
|
@@ -230,13 +230,13 @@ If instead you see:
|
|
|
230
230
|
Run `eigenpal auth login`, or set EIGENPAL_API_KEY in your env.
|
|
231
231
|
```
|
|
232
232
|
|
|
233
|
-
That
|
|
233
|
+
That is an **auth** error — the URL works but your key is missing /
|
|
234
234
|
expired / scoped to the wrong tenant. Run `auth list` to see configured
|
|
235
235
|
profiles, or `auth use <name>` to switch.
|
|
236
236
|
|
|
237
237
|
## 8. Override a flaky step in eval mode
|
|
238
238
|
|
|
239
|
-
If a step calls an external system that
|
|
239
|
+
If a step calls an external system that is unreliable (an external API, a
|
|
240
240
|
private API, a rate-limited connector), short-circuit it for evals via
|
|
241
241
|
the example's `meta.json`:
|
|
242
242
|
|
|
@@ -106,7 +106,7 @@ paths; you do not need a separate evaluator for failure cases.
|
|
|
106
106
|
Pick the label that best matches the tone of the response.
|
|
107
107
|
```
|
|
108
108
|
|
|
109
|
-
`promptExtension` is the only required field — it
|
|
109
|
+
`promptExtension` is the only required field — it is appended to the harness
|
|
110
110
|
prompt and tells the judge what "good" looks like for this workflow. The
|
|
111
111
|
harness handles JSON parsing, retries, and feeds the judge `actual`,
|
|
112
112
|
`expected`, and example metadata. In `discrete` mode the judge MUST return
|
|
@@ -140,7 +140,7 @@ verbatim, same text the dashboard editor shows. The `: number` return-type annot
|
|
|
140
140
|
enforced at parse time. The dashboard seeds new evaluators with a
|
|
141
141
|
`WorkflowOutput` type alias derived from the workflow's `output:`
|
|
142
142
|
declaration, plus `Expected` / `Actual` aliases for compatibility. Plain
|
|
143
|
-
`unknown` works too if you
|
|
143
|
+
`unknown` works too if you do not want a typed signature.
|
|
144
144
|
|
|
145
145
|
The signature shape is **locked**:
|
|
146
146
|
|
|
@@ -151,7 +151,7 @@ The signature shape is **locked**:
|
|
|
151
151
|
and scored as 0).
|
|
152
152
|
|
|
153
153
|
`workflow validate` and `workflow evaluators validate` flag YAML where
|
|
154
|
-
the function fails to parse, the signature
|
|
154
|
+
the function fails to parse, the signature does not match, or the body
|
|
155
155
|
contains neither `return` nor `throw` — so a hand-edited config errors
|
|
156
156
|
before it reaches the worker.
|
|
157
157
|
|
|
@@ -224,7 +224,7 @@ eigenpal workflow experiment results <workflow-id> <batch-id> \
|
|
|
224
224
|
--format csv --out results.csv
|
|
225
225
|
```
|
|
226
226
|
|
|
227
|
-
The signed-URL pattern means large result sets
|
|
227
|
+
The signed-URL pattern means large result sets do not go through the CLI
|
|
228
228
|
process — `--out` writes the file directly from the server's storage.
|
|
229
229
|
|
|
230
230
|
## Stabilizing flaky judges
|
|
@@ -264,9 +264,9 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
264
264
|
| Field | Type | Required | Default | Description |
|
|
265
265
|
| --- | --- | --- | --- | --- |
|
|
266
266
|
| `numericTolerance` | number | no | `0.000001` | Maximum absolute difference between numeric values that still counts as equal. Use 1e-6 for floats, 0 for exact integer match. |
|
|
267
|
-
| `allowExtraFields` | boolean | no | `true` | When on, extra fields in the actual output
|
|
267
|
+
| `allowExtraFields` | boolean | no | `true` | When on, extra fields in the actual output do not fail the diff. Off = actual must match expected exactly. |
|
|
268
268
|
| `passThreshold` | number | no | `1` | Minimum diff score this evaluator must reach for a run to pass. 1.0 = perfect match required. |
|
|
269
|
-
| `paths` | array<string> | no | | Dot-paths to scope the diff to a subset of the expected tree. Empty = diff entire expected. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`. Use this when expected has noisy sections you
|
|
269
|
+
| `paths` | array<string> | no | | Dot-paths to scope the diff to a subset of the expected tree. Empty = diff entire expected. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`. Use this when expected has noisy sections you do not want to score. |
|
|
270
270
|
|
|
271
271
|
|
|
272
272
|
### `llm-judge` — LLM-as-judge scoring
|
|
@@ -286,7 +286,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
286
286
|
| Field | Type | Required | Default | Description |
|
|
287
287
|
| --- | --- | --- | --- | --- |
|
|
288
288
|
| `function` | string | yes | | Full TypeScript source for optional `type` aliases plus `function scoreScript(expected, actual): number { ... }`. Receives `expected` (the example's expected output) and `actual` (the workflow's actual output), returns a number in [0, 1] (the `: number` annotation is required). The `: number` return-type annotation is required and enforced at parse time. Throws are caught and scored as 0. |
|
|
289
|
-
| `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it
|
|
289
|
+
| `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it is killed and the run is marked failed. |
|
|
290
290
|
| `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
|
|
291
291
|
| `passThreshold` | number | no | `1` | Minimum script score this evaluator must reach for a run to pass. |
|
|
292
292
|
<!-- GENERATED:EVALUATOR_CATALOG END -->
|
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
> retries, no tracing, no credit metering, no tenant-default LLM, no
|
|
8
8
|
> strict-mode response_format) and assumed shell env state
|
|
9
9
|
> (`WORKER_LLM_*`) that real CLI users — who authenticate via
|
|
10
|
-
> `EIGENPAL_API_KEY` —
|
|
10
|
+
> `EIGENPAL_API_KEY` — do not have.
|
|
11
11
|
>
|
|
12
12
|
> Calling `eigenpal workflow step exec <type>` today exits 2 with a
|
|
13
13
|
> redirect message. The server-side replacement is tracked in EIG-104:
|
|
@@ -30,7 +30,7 @@ eigenpal workflow experiment status <workflow-id> <batchId> --watch
|
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
For `--example-id`, the human slug works too (`ex-01-koifer-97zf`),
|
|
33
|
-
not
|
|
33
|
+
not the `evx_…` id — added as part of the same iteration.
|
|
34
34
|
|
|
35
35
|
To inspect what a step actually produced (input / output / resolved
|
|
36
36
|
config), use:
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Step types
|
|
2
2
|
|
|
3
3
|
The catalog is generated from `STEP_SCHEMAS` in
|
|
4
|
-
`@eigenpal/types/src/workflow/step-configs.ts`. **
|
|
4
|
+
`@eigenpal/types/src/workflow/step-configs.ts`. **Do not memorize fields
|
|
5
5
|
— introspect.** The CLI reads the live schema:
|
|
6
6
|
|
|
7
7
|
```bash
|
|
@@ -106,7 +106,7 @@ Reference user-supplied schema fields the same way:
|
|
|
106
106
|
|
|
107
107
|
`control.parallel`, `control.parallel_map`, `control.foreach`, `control.if`,
|
|
108
108
|
and `control.block` contain nested steps. The auto-generated catalog below
|
|
109
|
-
|
|
109
|
+
cannot render that shape, so the YAML form lives here. Same for the scoping
|
|
110
110
|
rules — important because the runtime treats nested steps differently from
|
|
111
111
|
top-level ones, and getting the access path wrong silently returns
|
|
112
112
|
`undefined` (no error).
|
|
@@ -114,7 +114,7 @@ top-level ones, and getting the access path wrong silently returns
|
|
|
114
114
|
### `control.parallel` — independent branches
|
|
115
115
|
|
|
116
116
|
Runs each branch concurrently. Use when several pieces of work share input
|
|
117
|
-
but
|
|
117
|
+
but do not depend on each other.
|
|
118
118
|
|
|
119
119
|
```yaml
|
|
120
120
|
- name: enrich
|
|
@@ -242,7 +242,7 @@ to preserve intermediate outputs).
|
|
|
242
242
|
|
|
243
243
|
> ⚠️ Foreach does NOT isolate scopes — the last iteration's inner step
|
|
244
244
|
> outputs technically persist on the top-level `steps` map and `steps.<inner>.output.x`
|
|
245
|
-
> resolves at runtime. That
|
|
245
|
+
> resolves at runtime. That is an only-the-last-row hazard; the autocomplete
|
|
246
246
|
> hides these names and you should treat `steps.<foreach>.output.items[]`
|
|
247
247
|
> as the only correct access.
|
|
248
248
|
|
|
@@ -304,7 +304,7 @@ not visible — the block runs in an isolated scope.
|
|
|
304
304
|
`ai.classify` + `control.fail` together let a workflow reject bad inputs
|
|
305
305
|
fast with a typed HTTP-style status code that callers (and evals) can
|
|
306
306
|
match against. The classifier picks one label from a closed set; the
|
|
307
|
-
fail step terminates the run when the label is one you
|
|
307
|
+
fail step terminates the run when the label is one you do not want to
|
|
308
308
|
process.
|
|
309
309
|
|
|
310
310
|
```yaml
|
|
@@ -326,7 +326,7 @@ same per-page representation.
|
|
|
326
326
|
Description-writing tips (mirrors Reducto's guidance):
|
|
327
327
|
|
|
328
328
|
- Use the document's own terminology — a Slovak contract calls it `Príloha`,
|
|
329
|
-
not "annex". List the multilingual variants you
|
|
329
|
+
not "annex". List the multilingual variants you have seen.
|
|
330
330
|
- Mention stable visual cues if any — a centered title-page header, a
|
|
331
331
|
signature line, the start of a specific table.
|
|
332
332
|
- Be specific. "The collateral schedule table at the start of Príloha 2"
|
|
@@ -442,11 +442,11 @@ eigenpal workflow push --file workflow.yaml --workflow-id wf_… --bump patch
|
|
|
442
442
|
`--bump` requires `--workflow-id` (it bumps from the current server
|
|
443
443
|
version). `--set-version` works with or without `--workflow-id`. The
|
|
444
444
|
YAML's top-level `version:` field is the fallback when neither flag is
|
|
445
|
-
passed — server rejects pushes whose version
|
|
445
|
+
passed — server rejects pushes whose version does not strictly increase.
|
|
446
446
|
|
|
447
447
|
## Discover step + evaluator types
|
|
448
448
|
|
|
449
|
-
|
|
449
|
+
Do not memorize fields — introspect:
|
|
450
450
|
|
|
451
451
|
```bash
|
|
452
452
|
eigenpal workflow step-type list # full step catalog
|
|
@@ -456,7 +456,7 @@ eigenpal workflow evaluator-type get <type> # config schema for one ev
|
|
|
456
456
|
```
|
|
457
457
|
|
|
458
458
|
Both catalogs are generated from the canonical Zod schemas in
|
|
459
|
-
`@eigenpal/types`, so they
|
|
459
|
+
`@eigenpal/types`, so they are always in sync with what the deployment
|
|
460
460
|
supports.
|
|
461
461
|
|
|
462
462
|
<!-- GENERATED:WORKFLOW_REFERENCE START -->
|
|
@@ -15,7 +15,7 @@ Add more examples under `dataset/examples/<name>/`. Each example needs an
|
|
|
15
15
|
`input/arguments.json`; `expected/output.json` and `meta.json` are optional.
|
|
16
16
|
File arguments live in `input/<arg-name>/<filename>`.
|
|
17
17
|
|
|
18
|
-
When you
|
|
18
|
+
When you are ready to push this workflow to the server:
|
|
19
19
|
|
|
20
20
|
```bash
|
|
21
21
|
eigenpal workflow push --file workflow.yaml
|
package/package.json
CHANGED
|
@@ -1,9 +0,0 @@
|
|
|
1
|
-
# eigenpal completion
|
|
2
|
-
|
|
3
|
-
Print a shell completion script to stdout (supported shells: bash, zsh, fish). Pipe into the appropriate completion file for your shell — see the script header for install instructions.
|
|
4
|
-
|
|
5
|
-
### Arguments
|
|
6
|
-
|
|
7
|
-
| Name | Required | Variadic | Description |
|
|
8
|
-
| ------- | -------- | -------- | ----------- |
|
|
9
|
-
| `shell` | yes | no | |
|
|
@@ -1,9 +0,0 @@
|
|
|
1
|
-
# eigenpal git
|
|
2
|
-
|
|
3
|
-
Passthrough to git with organization remote auth and committer identity.
|
|
4
|
-
|
|
5
|
-
### Arguments
|
|
6
|
-
|
|
7
|
-
| Name | Required | Variadic | Description |
|
|
8
|
-
| --------- | -------- | -------- | ---------------------- |
|
|
9
|
-
| `gitArgs` | no | yes | Git arguments after -- |
|