@eigenpal/cli 0.4.16 → 0.5.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (12) hide show
  1. package/CHANGELOG.md +1 -0
  2. package/dist/cli.js +819 -45
  3. package/dist/skill/SKILL.md +273 -436
  4. package/dist/skill/reference/cli/agent.md +536 -98
  5. package/dist/skill/reference/cli/auth.md +5 -5
  6. package/dist/skill/reference/cli/skill.md +5 -5
  7. package/dist/skill/reference/cli/workflow.md +97 -76
  8. package/dist/skill/reference/dataset-format.md +4 -0
  9. package/dist/skill/reference/debugging.md +14 -0
  10. package/dist/skill/reference/evaluators.md +4 -0
  11. package/dist/skill/reference/workflow-yaml.md +8 -0
  12. package/package.json +1 -1
package/dist/skill/SKILL.md CHANGED
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: eigenpal
3
- description: Build, run, and iterate on Eigenpal workflows from the command line. The workflow engine, evaluators, dataset folders, and execution introspection are all controlled via `eigenpal` subcommands. The server is the source of truth — local files are inputs you push.
3
+ description: Build, run, and iterate on Eigenpal workflows and agents from the command line. Use `eigenpal workflow` for YAML workflows, datasets, evaluators, executions, and experiments. Use `eigenpal agent` for agent workspaces, executions, feedback, traces, files, and expected artifacts.
4
4
  license: Apache-2.0
5
5
  compatibility: Requires eigenpal CLI.
6
6
  metadata:
@@ -11,533 +11,370 @@ metadata:
11
11
 
12
12
  # Eigenpal
13
13
 
14
- Eigenpal is a workflow engine. A workflow is a DAG of steps
15
- (parsing, extraction, scripting, etc.). Iteration happens locally with the
16
- `eigenpal` CLI; **execution + datasets + experiments live on the server**.
14
+ Eigenpal has two major CLI surfaces:
17
15
 
18
- This skill is the schema reference. The CLI does the actual work.
16
+ - `eigenpal workflow` — build and operate YAML workflows: definitions,
17
+ datasets, evaluators, executions, experiments, versions, and schema
18
+ introspection.
19
+ - `eigenpal agent` — build and operate agent workspaces: agent files, triggers,
20
+ executions, traces, feedback, expected artifacts, and reruns.
19
21
 
20
- ## Iteration loop (end-to-end)
22
+ Use long command names in generated commands and documentation. They are more
23
+ readable and cost very little for agents to type. Some short aliases exist for
24
+ interactive users; understand them if you see them, but do not prefer them:
21
25
 
22
- ```
23
- 1. eigenpal init workflow my-extraction --template pdf-extraction
24
- 2. edit workflow.yaml + evaluators.yaml + dataset/examples/*/expected/output.json
25
- 3. eigenpal workflow validate # local: all three checks, no server contact
26
- 4. eigenpal workflow push --file workflow.yaml # creates wf_… on first push
27
- 5. eigenpal workflow execution run my-extraction sample-name # one-off run for debugging
28
- 6. eigenpal workflow evaluators push <workflow-id> --file evaluators.yaml
29
- 7. eigenpal workflow dataset push <workflow-id> --file dataset/ --mode replace
30
- 8. eigenpal workflow experiment run <workflow-id> # → { batchId, executionIds }
31
- 9. eigenpal workflow experiment status <workflow-id> <batch-id>
32
- 10. eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out r.csv
33
- ```
34
-
35
- > **`push` BEFORE `execution run`.** The CLI no longer ships YAML at run
36
- > time — only saved workflows execute. `execution run <workflow-id>` reads
37
- > the definition from the server, so any local edit to `workflow.yaml`
38
- > must be `push`ed first or the run will use the previous version. If
39
- > the workflow has never been pushed, `execution run` exits with
40
- > `Workflow "<arg>" not found on the server. Push it first.`
41
-
42
- ## Identifying a workflow
43
-
44
- Every command that takes `<workflow-id>` expects a **`wf_xxx` id** — the
45
- opaque, server-generated identifier printed by `workflow push` and
46
- `workflow list`. Stable across renames; this is the form to use in
47
- scripts and CI.
26
+ - `agent exec` = `agent execution`
27
+ - `agent execution fb` = `agent execution feedback`
28
+ - `agent execution artifact` = `agent execution artifacts`
29
+ - `workflow exec` = `workflow execution`
30
+ - `workflow exp` = `workflow experiment`
31
+ - `ls` = `list` anywhere in the CLI
32
+ - `diff` = `compare` anywhere in the CLI
48
33
 
49
- As a typing convenience the workflow's slug (the YAML's `name:` field)
50
- is also accepted — the CLI resolves it server-side. The resolver
51
- detects the form by the `wf_` prefix:
34
+ For exact flags and defaults, prefer the generated references:
52
35
 
53
- | Argument starts with… | Looked up via |
54
- | --------------------- | ------------------------------------------------- |
55
- | `wf_…` | `GET /api/v1/workflows/<id>` |
56
- | anything else | `GET /api/v1/workflows?name=<slug>` (exact match) |
36
+ - [`reference/cli/workflow.md`](reference/cli/workflow.md)
37
+ - [`reference/cli/agent.md`](reference/cli/agent.md)
57
38
 
58
- Workflow names are constrained to `[a-z0-9][a-z0-9_-]*` (lowercase
59
- letters, digits, `_`, `-`; 1–64 chars), so every slug is terminal-safe
60
- and url-safe by construction — no shell quoting required.
39
+ ## Pick The Surface
61
40
 
62
- ```bash
63
- # Both work; prefer the id form in scripts.
64
- eigenpal workflow execution run wf_abc123 sample-1
65
- eigenpal workflow execution run my-extraction sample-1
66
- ```
67
-
68
- If the workflow hasn't been pushed yet, the resolver fails fast:
69
- `Workflow "<arg>" not found on the server. Push it first, or run \`eigenpal workflow list\` to see what's available.`
41
+ Choose `workflow` when the work is about a workflow definition, dataset,
42
+ evaluator, experiment, or workflow execution.
70
43
 
71
- **Other id types — not interchangeable with workflow id:**
44
+ Choose `agent` when the work is about an agent workspace, agent file, trigger,
45
+ agent execution, trace, feedback entry, expected artifact, or rerun.
72
46
 
73
- - `workflow execution {get,watch,cancel,compare} <executionId>` — `exec_…`
74
- - `workflow experiment compare <batchA> <batchB>` — batch ids (`evb_…`)
75
- - `workflow dataset example {get,update,delete} <wf-id> <exampleId>` — second positional is the example id (`evx_…`)
47
+ Agents and workflows are related, but their iteration loops are different:
48
+ workflow experiments evaluate datasets with evaluators; agent workflows do not
49
+ currently expose that same experiment/evaluator loop through the CLI.
76
50
 
77
- ## Common recipes
78
-
79
- ### From zero — new workflow from a template
51
+ ## Workflow Iteration Loop
80
52
 
81
53
  ```bash
82
- eigenpal init workflow invoices --template pdf-extraction # scaffolds workflow.yaml + dataset/ + evaluators.yaml
54
+ # 1. Create or edit the local project.
55
+ eigenpal init workflow invoices --template pdf-extraction
83
56
  cd invoices
84
- eigenpal workflow validate # confirm scaffold is valid before pushing
85
- eigenpal workflow push --file workflow.yaml
86
- # → ✓ Created wf_abc123 (`eigenpal workflow list` to see it again)
87
- ```
88
-
89
- ### Author + validate a workflow
57
+ $EDITOR workflow.yaml
90
58
 
91
- ```bash
92
- # Pick the right step types BEFORE writing the workflow
59
+ # 2. Inspect schemas before guessing config fields.
93
60
  eigenpal workflow step-type list --search extract
94
- eigenpal workflow step-type get ai.extract | jq '.configSchema.properties'
61
+ eigenpal workflow step-type get ai.extract --json | jq '.configSchema.properties'
95
62
 
96
- # Validate locally (zod parse + step-type lookup, no server)
63
+ # 3. Validate and push the workflow definition.
97
64
  eigenpal workflow validate ./workflow.yaml
65
+ eigenpal workflow push --file workflow.yaml
98
66
 
99
- # Push (creates on first call; same name + new version appends to history)
100
- eigenpal workflow push --file workflow.yaml --workflow-id wf_abc123
101
-
102
- # Update an existing workflow with explicit / auto-bumped semver:
103
- eigenpal workflow push --file workflow.yaml --workflow-id wf_abc123 --set-version 2.0.0
104
- eigenpal workflow push --file workflow.yaml --workflow-id wf_abc123 --bump patch # patch | minor | major
105
- ```
67
+ # 4. Run one example while iterating.
68
+ eigenpal workflow execution run <workflow-id> <example-name>
69
+ eigenpal workflow execution watch <execution-id>
70
+ eigenpal workflow execution get <execution-id> --json
106
71
 
107
- The validator emits structured field-path errors with a hint pointing at
108
- `step-type get <type>` for the offending step. Follow the hint.
72
+ # 5. Manage the dataset and evaluators when ready to evaluate.
73
+ eigenpal workflow dataset validate ./dataset
74
+ eigenpal workflow dataset push <workflow-id> --file ./dataset --mode replace
75
+ eigenpal workflow evaluators validate ./evaluators.yaml
76
+ eigenpal workflow evaluators push <workflow-id> --file ./evaluators.yaml
109
77
 
110
- Version bumping rules (`--bump` vs `--set-version` vs YAML `version:`)
111
- live in [`reference/workflow-yaml.md`](reference/workflow-yaml.md#validate-before-pushing).
78
+ # 6. Run and inspect a batch experiment.
79
+ eigenpal workflow experiment run <workflow-id>
80
+ eigenpal workflow experiment watch <workflow-id> <batch-id>
81
+ eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out results.csv
82
+ ```
112
83
 
113
- ### Iterate fast on one step (no server, no DAG)
84
+ Workflow commands that take `<workflow-id>` accept the stable `wf_...` id. Many
85
+ also accept the workflow slug for convenience, but use the id in scripts.
114
86
 
115
- Tightest possible loop — local sandbox, milliseconds per run. Today
116
- covers `transform.script` and `ai.extract`. For any other step type,
117
- fall back to `workflow execution run` against the server.
87
+ Useful workflow ids are not interchangeable:
118
88
 
119
- ```bash
120
- eigenpal workflow step exec transform.script \
121
- --config-json '{"function":"function script(items) { return items.reduce((s,i)=>s+i.v,0); }"}' \
122
- --inputs items=@items.json
123
- # 3
124
- ```
89
+ - `<workflow-id>` identifies a workflow (`wf_...`).
90
+ - `<execution-id>` identifies one workflow execution.
91
+ - `<batch-id>` identifies one experiment batch.
92
+ - `<example-id>` identifies one dataset example.
125
93
 
126
- See [`reference/step-exec.md`](reference/step-exec.md) for the full
127
- grammar — inputs, config, output schema, exit codes.
94
+ ### Organize workflows into folders
128
95
 
129
- ### Cancel a long-running execution
96
+ Folders live on the server, not in `workflow.yaml`. `workflow push` and
97
+ `workflow pull` do not read or write folder placement. Use `workflow move`
98
+ to move workflows between dashboard folders:
130
99
 
131
100
  ```bash
132
- eigenpal workflow execution cancel exec_abc # TTY: silent; CI: needs --yes
133
- eigenpal workflow execution cancel exec_abc --yes # CI / pipes
101
+ eigenpal workflow move <workflow-id> --folder billing/invoices
102
+ eigenpal workflow move <workflow-id> --folder /
134
103
  ```
135
104
 
136
- The worker finishes the current step, then stops — the check happens
137
- between every step transition, including inside `control.foreach` /
138
- `control.parallel` / `control.block` bodies. Already-terminal
139
- executions (completed/failed/cancelled) exit 0 with an info line. Safe
140
- to retry.
105
+ `--folder /` means root.
106
+ Missing folders in the path are created automatically.
107
+ Default to creating workflows at the top level. Do not move a workflow into
108
+ a folder unless the user specifically asks for that placement.
141
109
 
142
- ### Compare two experiment batches (no workflow id needed)
110
+ ## Agent Iteration Loop
143
111
 
144
112
  ```bash
145
- eigenpal workflow experiment compare evb_old evb_new --regression-threshold 0.05
113
+ # 1. Find the agent and inspect its files.
114
+ eigenpal agent list
115
+ eigenpal agent file list <agent-id-or-slug>
116
+ eigenpal agent file get <agent-id-or-slug> agent/instructions.md --out instructions.md
117
+
118
+ # 2. Find executions that need attention.
119
+ eigenpal agent execution list <agent-id-or-slug> \
120
+ --feedback-status open \
121
+ --feedback-rating fail \
122
+ --since-last-resolved \
123
+ --include feedback,expected \
124
+ --compact
125
+
126
+ # 3. Pull the execution context locally.
127
+ eigenpal agent execution pull <agent-execution-id> --include all --out ./review/<agent-execution-id>
128
+ eigenpal agent execution artifacts list <agent-execution-id>
129
+ eigenpal agent execution trace <agent-execution-id> --out ./review/<agent-execution-id>/trace.jsonl
130
+
131
+ # 4. Edit and preview agent files.
132
+ $EDITOR instructions.md
133
+ eigenpal agent file diff <agent-id-or-slug> agent/instructions.md instructions.md
134
+ eigenpal agent file put <agent-id-or-slug> agent/instructions.md instructions.md --preview
135
+ eigenpal agent file put <agent-id-or-slug> agent/instructions.md instructions.md
136
+
137
+ # 5. Rerun and compare.
138
+ eigenpal agent execution rerun <agent-execution-id> --wait
139
+ eigenpal agent execution compare <new-agent-execution-id> \
140
+ --expected-from <source-agent-execution-id> \
141
+ --normalize-dates
142
+ eigenpal agent execution compare <new-agent-execution-id> \
143
+ --baseline-from <source-agent-execution-id> \
144
+ --normalize-dates
145
+
146
+ # 6. Resolve feedback once verified.
147
+ eigenpal agent execution feedback resolve <source-agent-execution-id> \
148
+ --message "Fixed and verified in <new-agent-execution-id>."
146
149
  ```
147
150
 
148
- Output is two stacked tables: a per-evaluator aggregate (rows per evaluator
149
- with mean Δ, regressions, improvements — sorted biggest mover first) followed
150
- by a per-(example, evaluator) detail table. No need to write a Python
151
- aggregator on top.
152
-
153
- ```bash
154
- # JSON exposes both layers — `summary.byEvaluator` for the rollup,
155
- # `rows` for the full per-example detail.
156
- eigenpal workflow experiment compare evb_old evb_new --json | jq '.summary.byEvaluator'
157
- ```
151
+ Agent commands commonly accept `<agent-id-or-slug>`. Agent execution commands
152
+ take an agent execution id. Execution pulls and comparisons write review
153
+ artifacts under `.eigenpal/artifacts/...` by default.
158
154
 
159
- Unlike its siblings, `compare` takes no `<workflow-id>` — the server resolves
160
- the owning workflow from each batch id. Both batches must live in the
161
- same workflow within your tenant. See [`reference/debugging.md`](reference/debugging.md#6-compare-two-experiment-batches)
162
- for sort flags + `--json` shape.
155
+ ## Workflow Recipes
163
156
 
164
- ### Author + validate evaluators
157
+ ### Author And Validate A Workflow
165
158
 
166
159
  ```bash
167
- # Inspect what evaluator types exist + their config shape
168
- cat packages/cli/src/skill/reference/evaluators.md # or read this skill's `reference/`
169
-
170
- # Validate locally (every evaluator entry parsed against its discriminated-union schema)
171
- eigenpal workflow evaluators validate ./evaluators.yaml
172
-
173
- # Push (overwrites the workflow's evaluator config)
174
- eigenpal workflow evaluators push <workflow-id> --file evaluators.yaml
160
+ eigenpal workflow step-type list
161
+ eigenpal workflow step-type get <step-type> --json
162
+ eigenpal workflow validate ./workflow.yaml
163
+ eigenpal workflow push --file workflow.yaml
175
164
  ```
176
165
 
177
- Always set `description:` on every evaluator entry. It's a short, plain-language
178
- sentence (≤500 chars; aim for one line) explaining what the evaluator measures
179
- and why it matters, written for a non-technical reviewer in the dashboard.
180
- Skip model names, thresholds, and dot-paths; those belong in `config`. See
181
- [`reference/evaluators.md`](reference/evaluators.md#writing-descriptions-for-stakeholders).
166
+ The validator reports structured field-path errors. If the error hints at a
167
+ step type, inspect that type with `eigenpal workflow step-type get <type>`.
182
168
 
183
- ### Build + validate + upload a dataset
169
+ ### Manage A Dataset
184
170
 
185
171
  ```bash
186
- # Folder layout: dataset/examples/<name>/{input/arguments.json, input/<file-arg>/<file>, expected/output.json, meta.json}
187
- eigenpal workflow dataset validate ./dataset # rejects bad folder names, arg-name collisions, etc.
188
-
189
- # Push. `replace` wipes server-side examples for this workflow first; `append` adds.
190
- eigenpal workflow dataset push <workflow-id> --file ./dataset --mode replace
191
- # Watch the `expectedSet` count in the final NDJSON `done` event — if 0, evals run un-graded.
172
+ eigenpal workflow dataset validate ./dataset
173
+ eigenpal workflow dataset push <workflow-id> --file ./dataset --mode append
174
+ eigenpal workflow dataset list <workflow-id>
175
+ eigenpal workflow dataset pull <workflow-id> --out dataset.zip
192
176
  ```
193
177
 
194
- To inspect or round-trip the server's current dataset:
195
-
196
- ```bash
197
- eigenpal workflow dataset list <workflow-id>
198
- eigenpal workflow dataset pull <workflow-id> --out current.zip # round-trip back to local
178
+ Dataset archives use this folder shape:
179
+
180
+ ```text
181
+ dataset/
182
+ └── examples/
183
+ └── <example-name>/
184
+ ├── input/
185
+ │ ├── arguments.json
186
+ │ └── <file-argument-name>/<file>
187
+ ├── expected/output.json
188
+ └── meta.json
199
189
  ```
200
190
 
201
- ### Edit one example without re-uploading the whole dataset
191
+ For one-off changes, use the per-example commands instead of re-uploading the
192
+ whole dataset:
202
193
 
203
194
  ```bash
204
- # Inspect one example end-to-end (triggerInput + expectedOutput + metadata):
205
- eigenpal workflow dataset example get <workflow-id> <example-id>
206
- eigenpal workflow dataset example get <workflow-id> <example-id> --json | jq '.expectedOutput'
207
-
208
- # Capture a corrected output as the new ground truth (the most common one):
209
- # 1. Look at what the workflow returned
210
- eigenpal workflow execution get exec_… --json | jq '.output.data' > /tmp/correct.json
211
- # 2. Patch that example's expected output in place
212
- eigenpal workflow dataset example update <workflow-id> <example-id> \
213
- --expected-file /tmp/correct.json
214
-
215
- # Add one new example to a known dataset:
216
- eigenpal workflow dataset example create <workflow-id> \
217
- --name missing-required-field \
218
- --input-json '{"language":"en"}' \
219
- --expected-file /tmp/expected.json \
220
- --annotation "edge case: required field absent from input"
221
-
222
- # Drop a bad example by id (CI requires --yes):
195
+ eigenpal workflow dataset example get <workflow-id> <example-id> --json
196
+ eigenpal workflow dataset example update <workflow-id> <example-id> --expected-file expected.json
197
+ eigenpal workflow dataset example create <workflow-id> --name edge-case --input-json '{"language":"en"}'
223
198
  eigenpal workflow dataset example delete <workflow-id> <example-id> --yes
224
199
  ```
225
200
 
226
- `example update` is a partial PATCH — any flag you omit is left alone. Pass
227
- `--annotation ""` to clear an annotation. For bulk changes, edit the local
228
- dataset folder and re-push with `dataset push --mode replace`.
229
-
230
- ### Run an experiment + collect results
201
+ ### Run And Compare Workflow Executions
231
202
 
232
203
  ```bash
233
- # Kick off — runs every example in the dataset against every evaluator.
234
- eigenpal workflow experiment run <workflow-id> # → { batchId, executionIds }
235
-
236
- # Or restrict to one example:
237
- eigenpal workflow experiment run <workflow-id> --example-id evx_…
204
+ eigenpal workflow execution run <workflow-id> <example-name>
205
+ eigenpal workflow execution watch <execution-id>
206
+ eigenpal workflow execution get <execution-id> --include input,output,error --json
207
+ eigenpal workflow execution list <workflow-id> --status failed --limit 10
208
+ eigenpal workflow execution compare <execution-id-a> <execution-id-b>
209
+ eigenpal workflow execution cancel <execution-id> --yes
238
210
  ```
239
211
 
240
- **Recommended:** use `experiment watch` — it polls until terminal AND auto-pulls
241
- results to disk in one command. Replaces the old "run → status (poll) → results"
242
- chain so agents don't have to write a custom poller, parse `executions[].status`,
243
- or chain `monitor + status + pull + score` shell logic.
212
+ ### Run And Compare Experiments
244
213
 
245
214
  ```bash
246
- # One command: watch until terminal, then write ./results-<batchId>.json on completion.
247
- eigenpal workflow experiment watch <workflow-id> <batch-id>
248
-
249
- # CSV output:
250
- eigenpal workflow experiment watch <workflow-id> <batch-id> --format csv
251
-
252
- # Custom destination (overrides the default ./results-<batchId>.<format>):
253
- eigenpal workflow experiment watch <workflow-id> <batch-id> --pull-on-complete ./out/r.json
254
-
255
- # Just watch, skip the pull (e.g. you only need the live tick + exit code):
256
- eigenpal workflow experiment watch <workflow-id> <batch-id> --no-pull
215
+ eigenpal workflow experiment run <workflow-id>
216
+ eigenpal workflow experiment status <workflow-id> <batch-id> --json
217
+ eigenpal workflow experiment watch <workflow-id> <batch-id> --format json
218
+ eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out results.csv
219
+ eigenpal workflow experiment compare <batch-id-a> <batch-id-b> --regression-threshold 0.05
257
220
  ```
258
221
 
259
- Exit codes match `experiment status --watch`: `0` clean, `1` any
260
- failed/cancelled/rejected execution, `2` `--max-wait` deadline reached
261
- (30 min default — re-run to keep watching).
262
-
263
- ### Monitoring scripts — use `--short` and the JSON rollup
222
+ `workflow experiment compare` resolves the owning workflow from the batch ids;
223
+ both batches must belong to the same workflow.
264
224
 
265
- When a script needs to spot-check a batch (no waiting), prefer one of the two
266
- machine-readable paths instead of folding `executions[].status` yourself.
225
+ ## Agent Recipes
267
226
 
268
- ```bash
269
- # Single line on stdout, awk-friendly:
270
- eigenpal workflow experiment status <wf> <batch> --short
271
- # → 6/6 done failed=0 cancelled=0 rejected=0 (or "in-progress" while pending)
272
-
273
- # Parse with awk:
274
- DONE_STATE=$(eigenpal workflow experiment status <wf> <batch> --short | awk '{print $2}')
275
- [ "$DONE_STATE" = "done" ] || exit 0 # not terminal yet, try again later
276
- ```
227
+ ### Inspect And Edit Agent Files
277
228
 
278
229
  ```bash
279
- # JSON includes a top-level `summary` rollup so you don't count statuses by hand:
280
- eigenpal workflow experiment status <wf> <batch> --json | jq '.summary'
281
- # → { total: 6, terminal: 6, complete: true,
282
- # completedCount: 5, failedCount: 1,
283
- # cancelledCount: 0, rejectedCount: 0,
284
- # runningCount: 0, pendingCount: 0 }
285
-
286
- # `summary.complete` is the canonical poll-completion flag.
230
+ eigenpal agent list
231
+ eigenpal agent file list <agent-id-or-slug>
232
+ eigenpal agent file list <agent-id-or-slug> --prefix agent/knowledge
233
+ eigenpal agent file get <agent-id-or-slug> agent/instructions.md --out instructions.md
234
+ eigenpal agent file diff <agent-id-or-slug> agent/instructions.md instructions.md
235
+ eigenpal agent file put <agent-id-or-slug> agent/instructions.md instructions.md --preview
236
+ eigenpal agent file put <agent-id-or-slug> agent/instructions.md instructions.md
287
237
  ```
288
238
 
289
- `--short` and `--json` are mutually exclusive with `--watch` — for live
290
- streaming use `experiment watch` instead.
239
+ Use `--preview` before writing substantial changes; it shows how the target file
240
+ would change.
291
241
 
292
- If you need the raw eval-results export after watching with `--no-pull`, or
293
- to refetch results outside a watch loop, fall back to:
242
+ ### Inspect Agent Executions
294
243
 
295
244
  ```bash
296
- eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out r.csv
245
+ eigenpal agent execution list <agent-id-or-slug> --status failed --limit 10
246
+ eigenpal agent execution list <agent-id-or-slug> --feedback-rating fail --include feedback,expected
247
+ eigenpal agent execution get <agent-execution-id> --include feedback,expected,files,trace,issues --json
248
+ eigenpal agent execution pull <agent-execution-id> --include all
249
+ eigenpal agent execution artifacts list <agent-execution-id>
297
250
  ```
298
251
 
299
- ### Long documents — split → chunk → extract
300
-
301
- The canonical chain when a document is too long for one `ai.extract` call (loan
302
- contracts, SaaS agreements, RFPs). Each step preserves source `pageIndex` so
303
- the final extracted fields trace back to the originating page.
304
-
305
- ```yaml
306
- - name: parse
307
- type: ai.parse
308
- with: { input: "{{ input.document }}" }
309
-
310
- # LLM-driven section split — each split carries page_range + confidence
311
- - name: sections
312
- type: ai.split
313
- with:
314
- input: "{{ steps.parse.output }}"
315
- sections:
316
- - { name: header, description: "Title page + parties block" }
317
- - { name: priloha2, description: "Príloha 2 odkladacie podmienky", required: true }
318
-
319
- # Optional: chunk a long section before extraction (regex-anchored, keeps page refs)
320
- - name: chunks
321
- type: transform.text-chunker
322
- with:
323
- input: "{{ steps.sections.output.splits[1] }}" # priloha2 split
324
- maxChars: 8000
325
- overlap: 500
326
- splitOn: ['(?:^|\\n)\\s*\\d+\\.\\d+\\s+', '\\n\\n+', '\\n']
327
-
328
- # Deterministic field extraction — counterpart to ai.extract; matches carry _evidence.pageIndex
329
- - name: header_fields
330
- type: transform.regex-extract
331
- with:
332
- input: "{{ steps.parse.output }}"
333
- fields:
334
- contractNumber:
335
- pattern: 'č\\.\\s*(\\d{1,4}/\\w+/\\d{4})'
336
- normalize: strip-spaces
337
- signatureDate:
338
- pattern: 'uzavretá\\s+dňa\\s+(\\d{1,2})\\.(\\d{1,2})\\.(\\d{4})'
339
- format: iso-date
340
- currency:
341
- pattern: '\\b(EUR|USD|CZK)\\b'
342
- default: EUR
343
- flags: i
344
- ```
252
+ Use `--compact` on execution lists when you only need triage rows.
345
253
 
346
- Discovery from the CLI:
254
+ ### Download Agent Traces
347
255
 
348
256
  ```bash
349
- eigenpal workflow step-type get ai.split | jq '.configSchema.properties'
350
- eigenpal workflow step-type get transform.text-chunker | jq '.configSchema.properties'
351
- eigenpal workflow step-type get transform.regex-extract | jq '.configSchema.properties'
257
+ eigenpal agent execution trace <agent-execution-id> --out trace.jsonl
258
+ eigenpal agent execution trace <agent-execution-id> | jq -r 'select(.toolName? or .tool_name? or .tool?)'
259
+ eigenpal agent execution trace <agent-execution-id> | rg 'error|tool'
352
260
  ```
353
261
 
354
- Pick the right operator for the job:
355
-
356
- | Goal | Operator |
357
- | ---------------------------------------- | ------------------------- |
358
- | Extract structured data with an LLM | `ai.extract` |
359
- | Pull fields by regex (no LLM) | `transform.regex-extract` |
360
- | Split a doc into named sections (LLM) | `ai.split` |
361
- | Cut text by regex boundary + overlap | `transform.text-chunker` |
262
+ `trace` streams the raw JSONL to stdout by default. Use shell tools such as
263
+ `jq`, `rg`, or `awk` for filtering; use `--out` when you want to save the file.
362
264
 
363
- ### Debug a failing workflow
265
+ ### Manage Agent Feedback And Expected Artifacts
364
266
 
365
267
  ```bash
366
- # 1. Re-run a single example against the saved workflow (must be pushed first).
367
- # <workflow-id> accepts a `wf_…` id (preferred for scripts) or the slug.
368
- eigenpal workflow execution run <workflow-id> <example-name>
268
+ eigenpal agent execution feedback get <agent-execution-id>
269
+ eigenpal agent execution feedback update <agent-execution-id> \
270
+ --status open \
271
+ --rating fail \
272
+ --message "Expected the filing date to be extracted."
273
+ eigenpal agent execution feedback resolve <agent-execution-id> \
274
+ --message "Fixed and verified."
275
+ eigenpal agent execution feedback clear <agent-execution-id> --yes
276
+
277
+ eigenpal agent execution expected list <agent-execution-id>
278
+ eigenpal agent execution expected upload <agent-execution-id> --file expected.json --name expected.json
279
+ eigenpal agent execution expected copy-output <agent-execution-id> --output-file result.json --name expected.json
280
+ eigenpal agent execution expected pull <agent-execution-id> --out expected/
281
+ ```
369
282
 
370
- # 2. Or pull a recorded failure from the server
371
- eigenpal workflow execution list <workflow-id> --status failed --limit 5
372
- eigenpal workflow execution get exec_… # full payload, all steps
373
- eigenpal workflow execution get exec_… --step extract --include input,output,error
283
+ Feedback is attached to one execution. Expected artifacts are the references
284
+ used when comparing future runs.
374
285
 
375
- # 3. Compare two runs (e.g. before/after a workflow change)
376
- eigenpal workflow execution compare <exec-a> <exec-b>
286
+ ### Rerun And Compare Agent Executions
377
287
 
378
- # 4. Wait for a kicked-off run to finish (use this whenever you need
379
- # to block until terminal — adaptive polling, 30-min auto-detach)
380
- eigenpal workflow execution watch exec_…
288
+ ```bash
289
+ eigenpal agent execution rerun <agent-execution-id> --wait
290
+ eigenpal agent execution compare <new-agent-execution-id> --expected-from <source-agent-execution-id>
291
+ eigenpal agent execution compare <new-agent-execution-id> --baseline-from <source-agent-execution-id>
292
+ eigenpal agent execution compare <new-agent-execution-id> \
293
+ --expected-from <source-agent-execution-id> \
294
+ --normalize-dates \
295
+ --fail-on-diff
381
296
  ```
382
297
 
383
- See [`reference/debugging.md`](reference/debugging.md) for common failure
384
- modes (`validation_failed`, `template_resolution_failed`, `script_timeout`,
385
- `step_type_unknown`) and how to skip flaky external steps via
386
- `meta.json` overrides.
298
+ Use `--expected-from` to compare a new output against stored expected artifacts.
299
+ Use `--baseline-from` to compare a new output against another execution's actual
300
+ output.
387
301
 
388
- ## Output convention
302
+ ## Output And Exit Codes
389
303
 
390
- Every command splits its output:
391
-
392
- - **stdout** = data — JSON payloads, table rows, file contents from
393
- `pull` / `experiment results`. Pipe-safe.
394
- - **stderr** = status — `✓` / `✗` / `ℹ` / `!` lines. Silenced by
395
- `-q` / `--quiet` (errors and warnings always fire; stdout never
396
- silenced).
397
-
398
- The `--json` flag (on every `list` / `get` / mutating command) flips
399
- the human-readable table on stdout to the raw server payload — pipe
400
- through real `jq` to project:
304
+ Most `list`, `get`, and mutating commands support `--json`. Prefer JSON when a
305
+ script or agent needs to inspect the result with `jq`.
401
306
 
402
307
  ```bash
403
- eigenpal workflow execution list wf_abc --json | jq '.data[0].id'
404
- eigenpal workflow execution get exec_abc --json | jq '.stepExecutions[].status'
308
+ eigenpal workflow execution list <workflow-id> --json | jq '.data[0].id'
309
+ eigenpal agent execution list <agent-id-or-slug> --json | jq '.executions[0].id'
405
310
  ```
406
311
 
407
- ### Exit codes
312
+ General exit-code convention:
408
313
 
409
314
  | Code | Meaning |
410
315
  | --- | --- |
411
316
  | 0 | Success. |
412
- | 1 | Runtime error — network, server 5xx, command-specific failure (e.g. `experiment status` (with or without `--watch`) reports a terminal batch where at least one execution failed or was cancelled). |
413
- | 2 | Misuse or recoverable failure — bad flag, `step exec --output-schema` violation, `execution watch --max-wait` deadline, `step exec` against an unsupported step type. |
414
-
415
- Exit 2 means "fix your invocation and re-run." Exit 1 means something
416
- broke upstream — check the error.
417
-
418
- ## Where the schemas live
419
-
420
- The CLI ships JSON Schema for every step type and every evaluator type.
421
- Don't guess fields — introspect:
422
-
423
- ```bash
424
- eigenpal workflow step-type list # full step catalog
425
- eigenpal workflow step-type get ai.extract # config + output schema
426
- eigenpal workflow evaluator-type list # full evaluator catalog
427
- eigenpal workflow evaluator-type get llm-judge # config schema
428
- ```
429
-
430
- For workflow / evaluators / dataset shape, see this skill's `reference/`:
431
-
432
- - [`reference/workflow-yaml.md`](reference/workflow-yaml.md) — top-level shape, triggers, inputs, steps, output, template expressions
433
- - [`reference/step-types.md`](reference/step-types.md) — step-type catalog + when to reach for each
434
- - [`reference/step-exec.md`](reference/step-exec.md) — local single-step iteration (`workflow step exec`) — inputs, config, output schema, exit codes
435
- - [`reference/dataset-format.md`](reference/dataset-format.md) — `dataset/examples/<name>/{input,expected,meta}` folder convention
436
- - [`reference/evaluators.md`](reference/evaluators.md) — evaluators.yaml schema + judge patterns
437
- - [`reference/debugging.md`](reference/debugging.md) — execution introspection + common failures
438
- - [`reference/cli/`](reference/cli/) — full flag reference per command (autogenerated from the live Commander tree, never lies)
439
-
440
- ## Validation errors carry hints
441
-
442
- Every server-side validation failure returns a structured envelope:
443
-
444
- ```json
445
- {
446
- "issues": [
447
- { "field": "steps.2.config.passThreshold",
448
- "message": "Required, got undefined", "code": "invalid_value" }
449
- ],
450
- "hint": "Failing step is type `eval.llm-judge`. Run `eigenpal workflow step-type get eval.llm-judge` to inspect its config schema."
451
- }
317
+ | 1 | Runtime or command failure. |
318
+ | 2 | Invalid invocation, unsupported option, or timeout/deadline condition. |
319
+
320
+ Status and progress messages go to stderr. Data intended for piping goes to
321
+ stdout.
322
+
323
+ ## Schema And Reference Files
324
+
325
+ Workflow schemas:
326
+
327
+ - [`reference/workflow-yaml.md`](reference/workflow-yaml.md) — workflow file shape
328
+ - [`reference/step-types.md`](reference/step-types.md) — step-type catalog
329
+ - [`reference/step-exec.md`](reference/step-exec.md) — single-step execution command
330
+ - [`reference/dataset-format.md`](reference/dataset-format.md) — dataset folder format
331
+ - [`reference/evaluators.md`](reference/evaluators.md) — evaluator configuration
332
+ - [`reference/debugging.md`](reference/debugging.md) — workflow execution debugging
333
+
334
+ CLI command references:
335
+
336
+ - [`reference/cli/workflow.md`](reference/cli/workflow.md)
337
+ - [`reference/cli/agent.md`](reference/cli/agent.md)
338
+ - [`reference/cli/auth.md`](reference/cli/auth.md)
339
+ - [`reference/cli/status.md`](reference/cli/status.md)
340
+
341
+ Use `eigenpal --help` for the grouped command overview and
342
+ `eigenpal <command> --help` for live command help.
343
+
344
+ ## CLI Surface
345
+
346
+ ```text
347
+ eigenpal
348
+ ├── auth
349
+ │ ├── login
350
+ │ ├── list
351
+ │ └── use
352
+ ├── status
353
+ ├── workflow
354
+ │ ├── list
355
+ │ ├── pull
356
+ │ ├── push
357
+ │ ├── validate
358
+ │ ├── move
359
+ │ ├── clear-local
360
+ │ ├── dataset
361
+ │ ├── evaluators
362
+ │ ├── execution
363
+ │ ├── experiment
364
+ │ ├── versions
365
+ │ ├── step-type
366
+ │ └── evaluator-type
367
+ ├── agent
368
+ │ ├── list
369
+ │ ├── get
370
+ │ ├── push
371
+ │ ├── pull
372
+ │ ├── file
373
+ │ ├── trigger
374
+ │ ├── execution
375
+ │ └── experiment
376
+ └── skill
377
+ ├── install
378
+ ├── uninstall
379
+ └── list
452
380
  ```
453
-
454
- The CLI renders this with the field column aligned and the hint inline —
455
- follow the hint, the schema lookup is one command away.
456
-
457
- ## Don't reach for
458
-
459
- - `eval-local` / similar local-mirror commands — they don't exist. Drive
460
- the dataset by editing the local folder and pushing with
461
- `workflow dataset push --mode replace`, OR use the per-example CRUD
462
- (`dataset example {create,update,delete,get}`) when you only need to
463
- flip one row.
464
- - The `manifest.json` dataset format — legacy, the import endpoint rejects
465
- it. Always use the folder convention.
466
-
467
- ## CLI surface
468
-
469
- ```bash
470
- # Auth + tenant switching
471
- eigenpal auth login # add a profile (or update existing tenant's)
472
- eigenpal auth list # see all profiles, current marked ●
473
- eigenpal auth use [profile] # switch active (interactive picker if no arg)
474
- eigenpal status # active tenant, user, key id, workflow count
475
-
476
- # Definition
477
- eigenpal workflow list
478
- eigenpal workflow push --file workflow.yaml [--workflow-id <workflow-id>]
479
- eigenpal workflow pull <workflow-id>
480
- eigenpal workflow validate [path]
481
-
482
- # Evaluators
483
- eigenpal workflow evaluators push <workflow-id> --file evaluators.yaml
484
- eigenpal workflow evaluators pull <workflow-id>
485
- eigenpal workflow evaluators validate [path]
486
-
487
- # Dataset
488
- eigenpal workflow dataset push <workflow-id> --file dataset/ [--mode {append|replace}] # default: append
489
- eigenpal workflow dataset pull <workflow-id> --out dataset.zip
490
- eigenpal workflow dataset list <workflow-id>
491
- eigenpal workflow dataset validate [path]
492
- eigenpal workflow dataset example create <workflow-id> --name <n> [--input-file] [--expected-file] [--annotation]
493
- eigenpal workflow dataset example update <workflow-id> <example-id> [--name] [--input-file] [--expected-file] [--annotation] [--row-order]
494
- eigenpal workflow dataset example delete <workflow-id> <example-id> --yes
495
- eigenpal workflow dataset example get <workflow-id> <example-id> # full row + metadata
496
-
497
- # All-in-one validation against the templated project layout
498
- eigenpal workflow validate # ./workflow.yaml + ./evaluators.yaml + ./dataset/
499
-
500
- # Local artifacts
501
- eigenpal workflow clear-local # delete local execution artifacts (no server impact)
502
-
503
- # Experiments (batch eval runs)
504
- eigenpal workflow experiment run <workflow-id> [--example-id <id> ...]
505
- eigenpal workflow experiment status <workflow-id> <batch-id> [--watch] [--short] [--json] # exit 1 on terminal w/ failures, 2 on --max-wait. --json includes top-level `summary` rollup.
506
- eigenpal workflow experiment watch <workflow-id> <batch-id> [--format csv|json] [--pull-on-complete <path>] [--no-pull] # poll + auto-pull results in one shot
507
- eigenpal workflow experiment results <workflow-id> [batch-id] --format {csv|json} --out r.csv
508
- eigenpal workflow experiment list <workflow-id>
509
- eigenpal workflow experiment compare <batch-a> <batch-b> [--regression-threshold] [--sort] # no --workflow-id; prints per-evaluator aggregate + per-row table
510
-
511
- # Step-type / evaluator-type introspection
512
- eigenpal workflow step-type list
513
- eigenpal workflow step-type get <type>
514
- eigenpal workflow evaluator-type list
515
- eigenpal workflow evaluator-type get <type>
516
-
517
- # Local single-step iteration (no server)
518
- eigenpal workflow step exec <type> [--config-json | --config-file] [--inputs k=v...] [--output-schema]
519
-
520
- # Execution (one-off run + read-side debugging + cancellation)
521
- eigenpal workflow execution run <workflow-id> [examples...]
522
- eigenpal workflow execution get <exec-id>
523
- eigenpal workflow execution list <workflow-id> [--status running|failed|completed]
524
- eigenpal workflow execution watch <exec-id> # 2s/5s adaptive polling
525
- eigenpal workflow execution compare <exec-a> <exec-b>
526
- eigenpal workflow execution cancel <exec-id> # tells the worker to stop between steps; safe to retry
527
-
528
- # Versions
529
- eigenpal workflow versions list <workflow-id> # pushed semver history (newest first)
530
- eigenpal workflow versions restore <workflow-id> <version-id> # re-activate a prior version as the live one
531
-
532
- # Tooling
533
- eigenpal skill install # interactive multiselect for Claude Code, Cursor, …
534
- eigenpal skill uninstall [toolIds...] # name one or more tools, or pass --all
535
- eigenpal skill list # show what's installed where
536
- ```
537
-
538
- `eigenpal --help` for the grouped overview, `eigenpal <command> --help`
539
- for any specific command. The full per-command flag reference is also
540
- bundled at [`reference/cli/`](reference/cli/) (autogenerated from the
541
- live Commander tree, so it never lies). Read those when you need the
542
- exact spelling / default / data type for a flag without round-tripping
543
- to `--help`.