@eigenpal/cli 0.10.6 → 0.10.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -0
- package/dist/cli.js +381 -7280
- package/dist/skill/SKILL.md +20 -3
- package/dist/skill/reference/evaluators.md +4 -4
- package/package.json +1 -1
package/dist/skill/SKILL.md
CHANGED
|
@@ -20,7 +20,10 @@ appear in user examples, but do not prefer them.
|
|
|
20
20
|
experiments, versions, and step schemas.
|
|
21
21
|
- `eigenpal agents` manages Git-backed agent source packages, shared resources,
|
|
22
22
|
releases, sync, secrets, datasets, and live file inspection.
|
|
23
|
-
- `eigenpal run` starts a workflow or agent run from a target.
|
|
23
|
+
- `eigenpal run` starts a single workflow or agent run from a target. This is the
|
|
24
|
+
command for running something once and reading its output, including
|
|
25
|
+
API-triggered workflows. (`eigenpal workflow experiment` is a separate batch run
|
|
26
|
+
over a whole dataset, not the single-run path.)
|
|
24
27
|
- `eigenpal runs` inspects, watches, compares, reruns, cancels, and downloads
|
|
25
28
|
artifacts for workflow or agent runs.
|
|
26
29
|
|
|
@@ -46,12 +49,26 @@ eigenpal workflow step-type get ai.extract --json | jq '.configSchema.properties
|
|
|
46
49
|
|
|
47
50
|
# Validate and push.
|
|
48
51
|
eigenpal workflow validate ./workflow.yaml
|
|
52
|
+
# If the workflow has action.invoke-workflow steps, local validation cannot
|
|
53
|
+
# resolve their targets (sibling workflows live in the server DB). Add --online
|
|
54
|
+
# to check invoke targets, input types, and cycles before pushing.
|
|
55
|
+
eigenpal workflow validate ./workflow.yaml --online
|
|
49
56
|
eigenpal workflow push --file workflow.yaml
|
|
50
57
|
|
|
51
|
-
# Run
|
|
58
|
+
# Run with ad-hoc input and read the output (use this for API-triggered workflows).
|
|
59
|
+
# `--wait --json` polls to completion and prints the run with top-level `output`.
|
|
60
|
+
eigenpal run workflows.<workflow-id> --input-json '{"document":"..."}' --wait --json | jq '.output'
|
|
61
|
+
eigenpal run workflows.<workflow-id> --input-file document=./invoice.pdf --wait --json | jq '.output'
|
|
62
|
+
|
|
63
|
+
# Run one persisted example instead.
|
|
52
64
|
eigenpal run workflows.<workflow-id> --example <example-name>
|
|
65
|
+
|
|
66
|
+
# Inspect a run after the fact. `runs get --json` returns top-level
|
|
67
|
+
# `output`/`files`/`error` once the run is completed. Plain `run --wait`
|
|
68
|
+
# without `--json` only prints "Run <id> is completed", so pass `--json` or
|
|
69
|
+
# use `runs get` to see the output.
|
|
53
70
|
eigenpal runs watch <execution-id>
|
|
54
|
-
eigenpal runs get <execution-id> --json
|
|
71
|
+
eigenpal runs get <execution-id> --json | jq '.output'
|
|
55
72
|
|
|
56
73
|
# Push dataset and evaluators.
|
|
57
74
|
eigenpal workflow dataset validate ./dataset
|
|
@@ -249,7 +249,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
249
249
|
| Field | Type | Required | Default | Description |
|
|
250
250
|
| --- | --- | --- | --- | --- |
|
|
251
251
|
| `evaluators` | array<object> | no | `[]` | |
|
|
252
|
-
| `passThreshold` | number | no |
|
|
252
|
+
| `passThreshold` | number | no | | |
|
|
253
253
|
|
|
254
254
|
### Common entry fields (every evaluator type)
|
|
255
255
|
|
|
@@ -265,7 +265,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
265
265
|
| --- | --- | --- | --- | --- |
|
|
266
266
|
| `numericTolerance` | number | no | `0.000001` | Maximum absolute difference between numeric values that still counts as equal. Use 1e-6 for floats, 0 for exact integer match. |
|
|
267
267
|
| `allowExtraFields` | boolean | no | `true` | When on, extra fields in the actual output do not fail the diff. Off = actual must match expected exactly. |
|
|
268
|
-
| `passThreshold` | number | no |
|
|
268
|
+
| `passThreshold` | number | no | | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
|
|
269
269
|
| `paths` | array<string> | no | | Dot-paths to scope the diff to a subset of the expected tree. Empty = diff entire expected. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`. Use this when expected has noisy sections you do not want to score. |
|
|
270
270
|
|
|
271
271
|
|
|
@@ -276,7 +276,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
276
276
|
| `model` | string | no | | Which LLM grades the output. Falls back to the workspace's default LLM provider when unset. |
|
|
277
277
|
| `mode` | `"continuous"` \| `"discrete"` | no | `"continuous"` | Continuous = the LLM returns a free-form score in [0, 1]. Discrete = the LLM picks one of your labels and the score is looked up from the table. |
|
|
278
278
|
| `labels` | record<string, number> | no | | Allowed labels and the score each maps to. Required in discrete mode. The judge MUST return one of these labels; each score must be in [0, 1]. |
|
|
279
|
-
| `passThreshold` | number | no |
|
|
279
|
+
| `passThreshold` | number | no | | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
|
|
280
280
|
| `promptExtension` | string | yes | | Required. Evaluation criteria appended to the judge prompt — describe what makes a good answer for this workflow (tone, completeness, edge cases). Cannot be empty: an empty prompt yields the model inventing its own criteria. |
|
|
281
281
|
| `docLabel` | string | no | `"output"` | Name the judge uses to refer to the output in its prompt (e.g. 'invoice', 'extraction'). Default is 'output'. |
|
|
282
282
|
|
|
@@ -288,5 +288,5 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
|
|
|
288
288
|
| `function` | string | yes | | Full TypeScript source for optional `type` aliases plus `function scoreScript(expected, actual): number { ... }`. Receives `expected` (the example's expected output) and `actual` (the workflow's actual output), returns a number in [0, 1] (the `: number` annotation is required). The `: number` return-type annotation is required and enforced at parse time. Throws are caught and scored as 0. |
|
|
289
289
|
| `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it is killed and the run is marked failed. |
|
|
290
290
|
| `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
|
|
291
|
-
| `passThreshold` | number | no |
|
|
291
|
+
| `passThreshold` | number | no | | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
|
|
292
292
|
<!-- GENERATED:EVALUATOR_CATALOG END -->
|