@eigenpal/cli 0.7.2 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +101 -23
- package/README.md +8 -8
- package/dist/cli.js +9300 -9059
- package/dist/skill/SKILL.md +87 -415
- package/dist/skill/reference/cli/agents.md +26 -26
- package/dist/skill/reference/cli/rerun.md +1 -1
- package/dist/skill/reference/cli/run.md +1 -1
- package/dist/skill/reference/cli/runs.md +23 -2
- package/dist/skill/reference/cli/surface.md +1 -2
- package/dist/skill/reference/cli/workflow.md +10 -10
- package/dist/skill/reference/dataset-format.md +74 -241
- package/dist/skill/reference/debugging.md +5 -5
- package/dist/skill/reference/evaluators.md +10 -10
- package/dist/skill/reference/step-exec.md +2 -2
- package/dist/skill/reference/step-types.md +7 -7
- package/dist/skill/reference/workflow-yaml.md +4 -4
- package/dist/templates/blank/README.md +4 -3
- package/dist/templates/blank/evaluators.yaml +1 -1
- package/package.json +1 -1
- package/dist/skill/reference/cli/completion.md +0 -9
- package/dist/skill/reference/cli/git.md +0 -9
- /package/dist/templates/blank/dataset/examples/sample/{expected/output.json → expected.json} +0 -0
- /package/dist/templates/blank/dataset/examples/sample/{input/arguments.json → input.json} +0 -0
- /package/dist/templates/pdf-extraction/dataset/examples/sample-invoice/{expected/output.json → expected.json} +0 -0
- /package/dist/templates/pdf-extraction/dataset/examples/sample-invoice/{input/arguments.json → input.json} +0 -0
- /package/dist/templates/text-classification/dataset/examples/negative/{expected/output.json → expected.json} +0 -0
- /package/dist/templates/text-classification/dataset/examples/negative/{input/arguments.json → input.json} +0 -0
- /package/dist/templates/text-classification/dataset/examples/positive/{expected/output.json → expected.json} +0 -0
- /package/dist/templates/text-classification/dataset/examples/positive/{input/arguments.json → input.json} +0 -0
|
@@ -54,7 +54,7 @@ The grouped run payload includes:
|
|
|
54
54
|
substitution. Lets you see what the LLM actually got, what URL
|
|
55
55
|
HTTP actually hit, what items the script actually saw. Populated
|
|
56
56
|
for `ai.*` / `transform.*` / `action.*`. Null for `control.*`
|
|
57
|
-
(orchestration steps
|
|
57
|
+
(orchestration steps do not have a meaningful resolved config).
|
|
58
58
|
- `output` — what the step returned
|
|
59
59
|
- `error` — error message + code when `status: 'failed'`
|
|
60
60
|
- `durationMs`
|
|
@@ -108,7 +108,7 @@ no network or filesystem, so most timeouts are infinite-loop bugs.
|
|
|
108
108
|
|
|
109
109
|
### `step_type_unknown`
|
|
110
110
|
|
|
111
|
-
`workflow.yaml` references a `type:` that the deployment
|
|
111
|
+
`workflow.yaml` references a `type:` that the deployment does not know
|
|
112
112
|
about. Common after a typo or a step type that was renamed.
|
|
113
113
|
|
|
114
114
|
```bash
|
|
@@ -219,7 +219,7 @@ broken. If you see:
|
|
|
219
219
|
Set EIGENPAL_BASE_URL or pass --base-url <url>.
|
|
220
220
|
```
|
|
221
221
|
|
|
222
|
-
That
|
|
222
|
+
That is a **connection** error — the URL is wrong, the server is down,
|
|
223
223
|
or your network is blocked. The URL the CLI tried is echoed back so a
|
|
224
224
|
stale `EIGENPAL_BASE_URL` is immediately obvious.
|
|
225
225
|
|
|
@@ -230,13 +230,13 @@ If instead you see:
|
|
|
230
230
|
Run `eigenpal auth login`, or set EIGENPAL_API_KEY in your env.
|
|
231
231
|
```
|
|
232
232
|
|
|
233
|
-
That
|
|
233
|
+
That is an **auth** error — the URL works but your key is missing /
|
|
234
234
|
expired / scoped to the wrong tenant. Run `auth list` to see configured
|
|
235
235
|
profiles, or `auth use <name>` to switch.
|
|
236
236
|
|
|
237
237
|
## 8. Override a flaky step in eval mode
|
|
238
238
|
|
|
239
|
-
If a step calls an external system that
|
|
239
|
+
If a step calls an external system that is unreliable (an external API, a
|
|
240
240
|
private API, a rate-limited connector), short-circuit it for evals via
|
|
241
241
|
the example's `meta.json`:
|
|
242
242
|
|
|
@@ -54,7 +54,7 @@ finance) who never see the YAML or the judge prompt. Write it for them:
|
|
|
54
54
|
|
|
55
55
|
## Evaluator types
|
|
56
56
|
|
|
57
|
-
### `exact-diff` — JSON deep-diff against `expected
|
|
57
|
+
### `exact-diff` — JSON deep-diff against `expected.json`
|
|
58
58
|
|
|
59
59
|
```yaml
|
|
60
60
|
- name: invoice-fields
|
|
@@ -71,8 +71,8 @@ finance) who never see the YAML or the judge prompt. Write it for them:
|
|
|
71
71
|
is compared.
|
|
72
72
|
|
|
73
73
|
`exact-diff` branches on the example's expected shape: success-expected
|
|
74
|
-
examples (`expected
|
|
75
|
-
|
|
74
|
+
examples (`expected.json`) are diffed against the workflow's actual output, and
|
|
75
|
+
failure-expected examples (`expected.json` with `$error`, see
|
|
76
76
|
`reference/dataset-format.md`) are matched against the `control.fail`
|
|
77
77
|
envelope persisted to `executions.error`. The same evaluator handles both
|
|
78
78
|
paths; you do not need a separate evaluator for failure cases.
|
|
@@ -106,7 +106,7 @@ paths; you do not need a separate evaluator for failure cases.
|
|
|
106
106
|
Pick the label that best matches the tone of the response.
|
|
107
107
|
```
|
|
108
108
|
|
|
109
|
-
`promptExtension` is the only required field — it
|
|
109
|
+
`promptExtension` is the only required field — it is appended to the harness
|
|
110
110
|
prompt and tells the judge what "good" looks like for this workflow. The
|
|
111
111
|
harness handles JSON parsing, retries, and feeds the judge `actual`,
|
|
112
112
|
`expected`, and example metadata. In `discrete` mode the judge MUST return
|
|
@@ -140,7 +140,7 @@ verbatim, same text the dashboard editor shows. The `: number` return-type annot
|
|
|
140
140
|
enforced at parse time. The dashboard seeds new evaluators with a
|
|
141
141
|
`WorkflowOutput` type alias derived from the workflow's `output:`
|
|
142
142
|
declaration, plus `Expected` / `Actual` aliases for compatibility. Plain
|
|
143
|
-
`unknown` works too if you
|
|
143
|
+
`unknown` works too if you do not want a typed signature.
|
|
144
144
|
|
|
145
145
|
The signature shape is **locked**:
|
|
146
146
|
|
|
@@ -151,7 +151,7 @@ The signature shape is **locked**:
|
|
|
151
151
|
and scored as 0).
|
|
152
152
|
|
|
153
153
|
`workflow validate` and `workflow evaluators validate` flag YAML where
|
|
154
|
-
the function fails to parse, the signature
|
|
154
|
+
the function fails to parse, the signature does not match, or the body
|
|
155
155
|
contains neither `return` nor `throw` — so a hand-edited config errors
|
|
156
156
|
before it reaches the worker.
|
|
157
157
|
|
|
@@ -224,7 +224,7 @@ eigenpal workflow experiment results <workflow-id> <batch-id> \
|
|
|
224
224
|
--format csv --out results.csv
|
|
225
225
|
```
|
|
226
226
|
|
|
227
|
-
The signed-URL pattern means large result sets
|
|
227
|
+
The signed-URL pattern means large result sets do not go through the CLI
|
|
228
228
|
process — `--out` writes the file directly from the server's storage.
|
|
229
229
|
|
|
230
230
|
## Stabilizing flaky judges
|
|
@@ -264,9 +264,9 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
264
264
|
| Field | Type | Required | Default | Description |
|
|
265
265
|
| --- | --- | --- | --- | --- |
|
|
266
266
|
| `numericTolerance` | number | no | `0.000001` | Maximum absolute difference between numeric values that still counts as equal. Use 1e-6 for floats, 0 for exact integer match. |
|
|
267
|
-
| `allowExtraFields` | boolean | no | `true` | When on, extra fields in the actual output
|
|
267
|
+
| `allowExtraFields` | boolean | no | `true` | When on, extra fields in the actual output do not fail the diff. Off = actual must match expected exactly. |
|
|
268
268
|
| `passThreshold` | number | no | `1` | Minimum diff score this evaluator must reach for a run to pass. 1.0 = perfect match required. |
|
|
269
|
-
| `paths` | array<string> | no | | Dot-paths to scope the diff to a subset of the expected tree. Empty = diff entire expected. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`. Use this when expected has noisy sections you
|
|
269
|
+
| `paths` | array<string> | no | | Dot-paths to scope the diff to a subset of the expected tree. Empty = diff entire expected. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`. Use this when expected has noisy sections you do not want to score. |
|
|
270
270
|
|
|
271
271
|
|
|
272
272
|
### `llm-judge` — LLM-as-judge scoring
|
|
@@ -286,7 +286,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
286
286
|
| Field | Type | Required | Default | Description |
|
|
287
287
|
| --- | --- | --- | --- | --- |
|
|
288
288
|
| `function` | string | yes | | Full TypeScript source for optional `type` aliases plus `function scoreScript(expected, actual): number { ... }`. Receives `expected` (the example's expected output) and `actual` (the workflow's actual output), returns a number in [0, 1] (the `: number` annotation is required). The `: number` return-type annotation is required and enforced at parse time. Throws are caught and scored as 0. |
|
|
289
|
-
| `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it
|
|
289
|
+
| `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it is killed and the run is marked failed. |
|
|
290
290
|
| `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
|
|
291
291
|
| `passThreshold` | number | no | `1` | Minimum script score this evaluator must reach for a run to pass. |
|
|
292
292
|
<!-- GENERATED:EVALUATOR_CATALOG END -->
|
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
> retries, no tracing, no credit metering, no tenant-default LLM, no
|
|
8
8
|
> strict-mode response_format) and assumed shell env state
|
|
9
9
|
> (`WORKER_LLM_*`) that real CLI users — who authenticate via
|
|
10
|
-
> `EIGENPAL_API_KEY` —
|
|
10
|
+
> `EIGENPAL_API_KEY` — do not have.
|
|
11
11
|
>
|
|
12
12
|
> Calling `eigenpal workflow step exec <type>` today exits 2 with a
|
|
13
13
|
> redirect message. The server-side replacement is tracked in EIG-104:
|
|
@@ -30,7 +30,7 @@ eigenpal workflow experiment status <workflow-id> <batchId> --watch
|
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
For `--example-id`, the human slug works too (`ex-01-koifer-97zf`),
|
|
33
|
-
not
|
|
33
|
+
not the `evx_…` id — added as part of the same iteration.
|
|
34
34
|
|
|
35
35
|
To inspect what a step actually produced (input / output / resolved
|
|
36
36
|
config), use:
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Step types
|
|
2
2
|
|
|
3
3
|
The catalog is generated from `STEP_SCHEMAS` in
|
|
4
|
-
`@eigenpal/types/src/workflow/step-configs.ts`. **
|
|
4
|
+
`@eigenpal/types/src/workflow/step-configs.ts`. **Do not memorize fields
|
|
5
5
|
— introspect.** The CLI reads the live schema:
|
|
6
6
|
|
|
7
7
|
```bash
|
|
@@ -106,7 +106,7 @@ Reference user-supplied schema fields the same way:
|
|
|
106
106
|
|
|
107
107
|
`control.parallel`, `control.parallel_map`, `control.foreach`, `control.if`,
|
|
108
108
|
and `control.block` contain nested steps. The auto-generated catalog below
|
|
109
|
-
|
|
109
|
+
cannot render that shape, so the YAML form lives here. Same for the scoping
|
|
110
110
|
rules — important because the runtime treats nested steps differently from
|
|
111
111
|
top-level ones, and getting the access path wrong silently returns
|
|
112
112
|
`undefined` (no error).
|
|
@@ -114,7 +114,7 @@ top-level ones, and getting the access path wrong silently returns
|
|
|
114
114
|
### `control.parallel` — independent branches
|
|
115
115
|
|
|
116
116
|
Runs each branch concurrently. Use when several pieces of work share input
|
|
117
|
-
but
|
|
117
|
+
but do not depend on each other.
|
|
118
118
|
|
|
119
119
|
```yaml
|
|
120
120
|
- name: enrich
|
|
@@ -242,7 +242,7 @@ to preserve intermediate outputs).
|
|
|
242
242
|
|
|
243
243
|
> ⚠️ Foreach does NOT isolate scopes — the last iteration's inner step
|
|
244
244
|
> outputs technically persist on the top-level `steps` map and `steps.<inner>.output.x`
|
|
245
|
-
> resolves at runtime. That
|
|
245
|
+
> resolves at runtime. That is an only-the-last-row hazard; the autocomplete
|
|
246
246
|
> hides these names and you should treat `steps.<foreach>.output.items[]`
|
|
247
247
|
> as the only correct access.
|
|
248
248
|
|
|
@@ -304,7 +304,7 @@ not visible — the block runs in an isolated scope.
|
|
|
304
304
|
`ai.classify` + `control.fail` together let a workflow reject bad inputs
|
|
305
305
|
fast with a typed HTTP-style status code that callers (and evals) can
|
|
306
306
|
match against. The classifier picks one label from a closed set; the
|
|
307
|
-
fail step terminates the run when the label is one you
|
|
307
|
+
fail step terminates the run when the label is one you do not want to
|
|
308
308
|
process.
|
|
309
309
|
|
|
310
310
|
```yaml
|
|
@@ -339,7 +339,7 @@ process.
|
|
|
339
339
|
The synchronous run endpoint surfaces the `statusCode` as the HTTP
|
|
340
340
|
response status; async runs persist `{ code, message, step }` to
|
|
341
341
|
`executions.error` so the eval scorer can match against
|
|
342
|
-
`expected
|
|
342
|
+
`expected.json` with `$error` (see `reference/dataset-format.md`). When
|
|
343
343
|
`condition` is omitted, `control.fail` always fails when reached, so
|
|
344
344
|
compose with `control.if` for legacy gating.
|
|
345
345
|
|
|
@@ -526,7 +526,7 @@ Merge multiple named inputs into a single output
|
|
|
526
526
|
|
|
527
527
|
#### `transform.template` — Fill Template
|
|
528
528
|
|
|
529
|
-
Fill a DOCX template with data.
|
|
529
|
+
Fill a DOCX template with data from previous steps. Select a template in the workflow builder or provide a template ID from your workspace.
|
|
530
530
|
|
|
531
531
|
**Config** (in `step.with`):
|
|
532
532
|
|
|
@@ -326,7 +326,7 @@ same per-page representation.
|
|
|
326
326
|
Description-writing tips (mirrors Reducto's guidance):
|
|
327
327
|
|
|
328
328
|
- Use the document's own terminology — a Slovak contract calls it `Príloha`,
|
|
329
|
-
not "annex". List the multilingual variants you
|
|
329
|
+
not "annex". List the multilingual variants you have seen.
|
|
330
330
|
- Mention stable visual cues if any — a centered title-page header, a
|
|
331
331
|
signature line, the start of a specific table.
|
|
332
332
|
- Be specific. "The collateral schedule table at the start of Príloha 2"
|
|
@@ -442,11 +442,11 @@ eigenpal workflow push --file workflow.yaml --workflow-id wf_… --bump patch
|
|
|
442
442
|
`--bump` requires `--workflow-id` (it bumps from the current server
|
|
443
443
|
version). `--set-version` works with or without `--workflow-id`. The
|
|
444
444
|
YAML's top-level `version:` field is the fallback when neither flag is
|
|
445
|
-
passed — server rejects pushes whose version
|
|
445
|
+
passed — server rejects pushes whose version does not strictly increase.
|
|
446
446
|
|
|
447
447
|
## Discover step + evaluator types
|
|
448
448
|
|
|
449
|
-
|
|
449
|
+
Do not memorize fields — introspect:
|
|
450
450
|
|
|
451
451
|
```bash
|
|
452
452
|
eigenpal workflow step-type list # full step catalog
|
|
@@ -456,7 +456,7 @@ eigenpal workflow evaluator-type get <type> # config schema for one ev
|
|
|
456
456
|
```
|
|
457
457
|
|
|
458
458
|
Both catalogs are generated from the canonical Zod schemas in
|
|
459
|
-
`@eigenpal/types`, so they
|
|
459
|
+
`@eigenpal/types`, so they are always in sync with what the deployment
|
|
460
460
|
supports.
|
|
461
461
|
|
|
462
462
|
<!-- GENERATED:WORKFLOW_REFERENCE START -->
|
|
@@ -12,10 +12,11 @@ eigenpal run workflows.__NAME__ --example sample
|
|
|
12
12
|
```
|
|
13
13
|
|
|
14
14
|
Add more examples under `dataset/examples/<name>/`. Each example needs an
|
|
15
|
-
`input
|
|
16
|
-
|
|
15
|
+
`input.json`; `expected.json` and `meta.json` are optional. File arguments live
|
|
16
|
+
under `input/` and are referenced from `input.json` with
|
|
17
|
+
`{ "$file": "input/<filename>" }`.
|
|
17
18
|
|
|
18
|
-
When you
|
|
19
|
+
When you are ready to push this workflow to the server:
|
|
19
20
|
|
|
20
21
|
```bash
|
|
21
22
|
eigenpal workflow push --file workflow.yaml
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Evaluators run after every workflow execution that's part of an eval batch.
|
|
2
|
-
# Comment them in once you have an `expected
|
|
2
|
+
# Comment them in once you have an `expected.json` output object to compare against.
|
|
3
3
|
#
|
|
4
4
|
# evaluators:
|
|
5
5
|
# - name: exact-match
|
package/package.json
CHANGED
|
@@ -1,9 +0,0 @@
|
|
|
1
|
-
# eigenpal completion
|
|
2
|
-
|
|
3
|
-
Print a shell completion script to stdout (supported shells: bash, zsh, fish). Pipe into the appropriate completion file for your shell — see the script header for install instructions.
|
|
4
|
-
|
|
5
|
-
### Arguments
|
|
6
|
-
|
|
7
|
-
| Name | Required | Variadic | Description |
|
|
8
|
-
| ------- | -------- | -------- | ----------- |
|
|
9
|
-
| `shell` | yes | no | |
|
|
@@ -1,9 +0,0 @@
|
|
|
1
|
-
# eigenpal git
|
|
2
|
-
|
|
3
|
-
Passthrough to git with organization remote auth and committer identity.
|
|
4
|
-
|
|
5
|
-
### Arguments
|
|
6
|
-
|
|
7
|
-
| Name | Required | Variadic | Description |
|
|
8
|
-
| --------- | -------- | -------- | ---------------------- |
|
|
9
|
-
| `gitArgs` | no | yes | Git arguments after -- |
|
/package/dist/templates/blank/dataset/examples/sample/{expected/output.json → expected.json}
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|