@eigenpal/cli 0.0.1 → 0.4.2

package/dist/skill/reference/dataset-format.md

@@ -0,0 +1,251 @@
+# Dataset folder format
+
+A dataset is a folder of named example folders. The folder structure
+itself is the manifest — there is no top-level `manifest.json`. The
+import endpoint **rejects** archives that contain one (legacy format).
+
+This is the format the CLI builds and ships to
+`eigenpal workflow dataset push <wf-id> --file dataset/`. After it lands
+on the server, the canonical per-row surface is
+`eigenpal workflow dataset example {get,create,update,delete}` (and
+`dataset list` for a table view) — the local folder is a pre-push
+staging area, not a synced mirror.
+
+## Required layout
+
+```
+dataset/
+└── examples/
+    ├── invoice-foo/
+    │   ├── input/
+    │   │   ├── arguments.json              REQUIRED — { language: "en" }
+    │   │   └── contract/                   file-argument folder
+    │   │       ├── Contract_2026.pdf
+    │   │       └── Appendix.pdf
+    │   ├── expected/
+    │   │   ├── output.json                 OPTIONAL — raw ground truth JSON
+    │   │   └── generated_invoice/          OPTIONAL — expected file output
+    │   │       └── Invoice.docx
+    │   └── meta.json                       OPTIONAL — { rowOrder?, annotation?, overrides? }
+    │
+    └── invoice-bar/
+        └── input/
+            └── arguments.json              { amount: 42 }
+```
+
+The `expected/` subfolder mirrors `input/` exactly — `output.json` is
+the scalar/object data ground truth, and any expected file outputs go
+in `expected/<docKey>/<file>` folders (one folder per output path).
+
+## Rules (every one is enforced server-side at import)
+
+- **Folder names** (example names, argument names, expected document
+  keys) must match `[a-z0-9][a-z0-9-_]*`.
+- **`input/arguments.json`** is REQUIRED for every example, even if it's
+  just `{}`. It must be a JSON object (not array, not literal).
+- **File-arguments** live in `input/<arg-name>/<filename>`. The
+  materialized shape on `triggerInput` is **driven by the workflow YAML's
+  `inputs[].type`**, not by the file count:
+
+  | workflow.yaml input                        | dataset folder            | `triggerInput.<name>` |
+  | ------------------------------------------ | ------------------------- | --------------------- |
+  | `type: file`                               | `input/<name>/<one-file>` | `{ fileId }`          |
+  | `type: array`, `items: { type: file }`     | `input/<name>/<file>...`  | `[{ fileId }, ...]`   |
+
+  Reference single-file inputs as `{{ input.<name> }}` (resolves to the
+  `{ fileId }` object the worker reads). Reference array-of-file inputs
+  as `{{ input.<name>[0] }}` etc. Dropping multiple binaries into an
+  `input/<name>/` folder for an input declared `type: file` is rejected
+  at import with `code: single_file_input_overpopulated` — either trim
+  to one file or change the workflow input to the array form.
+- **Argument-name collisions are rejected.** If `arguments.json` has
+  `{"contract": "..."}` AND `input/contract/file.pdf` exists, import
+  fails with `code: argument_name_collision`.
+- **`expected/output.json`** is OPTIONAL. When present, it must be a
+  JSON object that mirrors your workflow's `output:` shape 1:1 — no
+  envelope, no wrapper. A scalar/array at the top level is rejected
+  with `code: invalid_expected_output`.
+
+  > **Note:** the importer is idempotent and accepts both bare objects and
+  > pre-wrapped `{ data: { ... } }` forms; bare is preferred.
+
+  ```json
+  // workflow.yaml
+  // output:
+  //   invoiceNumber: '{{ steps.extract.output.invoiceNumber }}'
+  //   totalAmount:   '{{ steps.extract.output.totalAmount }}'
+  //
+  // → expected/output.json
+  {
+    "invoiceNumber": "INV-001",
+    "totalAmount": 1234.56
+  }
+  ```
+- **Expected file outputs** live in `expected/<docKey>/<filename>` —
+  symmetric with `input/<argName>/<file>`. Each `<docKey>` folder is
+  uploaded and becomes an `expectedDocuments[docKey]` entry server-side
+  (single file ⇒ `{fileId}`, multiple ⇒ `[{fileId}, …]`). The judge
+  compares by file presence + content, not byte-equal.
+- **`meta.json`** is OPTIONAL. Schema:
+  ```json
+  {
+    "rowOrder": 0,
+    "annotation": "free-form notes about this example",
+    "overrides": {
+      "steps": {
+        "<stepName>": { ...stepOutputObject }
+      }
+    }
+  }
+  ```
+  - `rowOrder` — sort order in the dashboard
+  - `annotation` — surfaced in the UI; useful for "why this case is
+    interesting" notes
+  - `overrides.steps.<stepName>` — short-circuits the named step,
+    returns the override as the step's output. Use for steps that
+    depend on external state (third-party APIs, private connectors) so the rest
+    of the example can still run.
+- **Path-traversal patterns** (`..`, leading `/`, backslashes, null
+  bytes) reject the whole archive.
+
+## How the row materializes
+
+For an example with:
+
+- `arguments.json`: `{"language": "en"}`
+- `input/contract/Contract_2026.pdf`
+- `input/contract/Appendix.pdf`
+- `expected/output.json`: `{"invoiceNumber": "INV-001"}`
+- `expected/generated_invoice/Invoice.docx`
+
+The server materializes:
+
+```json
+// eval_examples.triggerInput
+{
+  "language": "en",
+  "contract": [
+    { "fileId": "file_abc..." },
+    { "fileId": "file_def..." }
+  ]
+}
+
+// eval_examples.expectedOutput
+{
+  "data": { "invoiceNumber": "INV-001" },
+  "expectedDocuments": {
+    "generated_invoice": { "fileId": "file_ghi..." }
+  }
+}
+```
+
+Each file is uploaded to S3 with a fresh `fileId`; the original
+filename is preserved on the `files` row and shows up in the dashboard.
+The `expectedOutput` envelope (`{ data?, expectedDocuments? }`) is
+internal storage shape — your dataset folder stays clean (raw
+`output.json` + per-docKey folders).
+
+## Validate before pushing
+
+```bash
+eigenpal workflow validate dataset ./dataset
+```
+
+Reports per-example issues:
+
+```
+✗ dataset (./dataset) — 2 issues
+  examples/Invoice-Foo                   Folder name must be lowercase kebab/snake-case.
+  examples/foo/input/contract            Argument-name collision: "contract" appears in both arguments.json and as an input/ folder.
+```
+
+## Push
+
+```bash
+# Replace = wipes existing examples for the workflow, uploads the folder fresh.
+eigenpal workflow dataset push <wf-id> --file ./dataset --mode replace
+
+# Append = adds to whatever's already on the server.
+eigenpal workflow dataset push <wf-id> --file ./dataset --mode append
+```
+
+The endpoint streams progress as NDJSON. The terminal `done` event
+includes `{ created, expectedSet, … }`:
+
+- `created` — examples successfully persisted
+- `expectedSet` — how many of those carried an `expected/output.json`.
+  If `expectedSet < created`, your zip didn't include the `expected/`
+  folders — the dashboard's "Expected" tab will be blank and evals
+  will only run un-graded.
+
+## Editing a dataset on the server
+
+To inspect or round-trip what's currently on the server:
+
+```bash
+eigenpal workflow dataset list <wf-id>
+eigenpal workflow dataset pull <wf-id> --out current.zip
+# (or omit --out to stream the zip to stdout for piping)
+```
+
+For bulk changes — many examples at once, schema-level rework, file
+swaps — edit the local folder and re-push with
+`dataset push --mode replace`.
+
+For one-row tweaks — capturing a corrected output as the new GT,
+deleting one bad example, adding one example to a known dataset — use
+the per-example CRUD commands instead. They avoid the round-trip of
+re-zipping and re-uploading the whole folder.
+
+### Inspect one example end-to-end
+
+```bash
+eigenpal workflow dataset example get <wf-id> <example-id>
+eigenpal workflow dataset example get <wf-id> <example-id> --json | jq '.expectedOutput'
+```
+
+Pretty mode prints labelled `Inputs` / `Expected` / `Metadata` sections;
+`--json` returns the full row (`triggerInput`, `expectedOutput`,
+`annotation`, `rowOrder`, timestamps) for piping.
+
+### Capture a corrected output as the new ground truth
+
+```bash
+# 1. Look at what the workflow actually returned for the example
+eigenpal workflow execution get exec_… --json | jq '.output.data' > /tmp/correct.json
+
+# 2. Patch the example's expected output in place. PATCH semantics:
+#    every flag you omit is left alone.
+eigenpal workflow dataset example update <wf-id> <example-id> \
+  --expected-file /tmp/correct.json
+```
+
+`--expected-file -` reads JSON from stdin if you'd rather pipe. Pass
+empty string to clear an annotation: `--annotation ""`.
+
+### Add one new example
+
+```bash
+eigenpal workflow dataset example create <wf-id> \
+  --name missing-required-field \
+  --input-json '{"language":"en"}' \
+  --expected-file /tmp/expected.json \
+  --annotation "edge case: required field absent from input"
+```
+
+`--input-json` / `--expected-json` accept inline JSON literals; the
+`-file` variants read from disk (or `-` for stdin). They're mutually
+exclusive — pass one or the other, not both.
+
+File-arg uploads still go through `dataset push`; CRUD only handles
+scalar args + `expected/output.json`-style outputs.
+
+### Delete one bad example
+
+```bash
+eigenpal workflow dataset example delete <wf-id> <example-id> --yes
+```
+
+`--yes` is required for non-TTY shells (CI). Interactive shells may
+omit it; single-row deletes have a small blast radius compared to
+`dataset push --mode replace`, so no confirmation prompt fires.

package/dist/skill/reference/debugging.md

@@ -0,0 +1,280 @@
+# Debugging a failing workflow
+
+Three tools cover most cases:
+
+```bash
+eigenpal workflow execution get   <exec-id>           # full payload, step-by-step
+eigenpal workflow execution watch <exec-id>           # live status; 2s during transition / 5s steady
+eigenpal workflow execution compare <a> <b>           # side-by-side diff of two runs
+```
+
+## 1. Reproduce the failure
+
+```bash
+eigenpal workflow execution run <workflow-name> <example-name>
+```
+
+Streams a live step list as the run progresses. Bails to the prompt
+the moment a step transitions to `failed`.
+
+If the workflow is server-stable and the failure is on a real
+historical execution, skip the rerun and pull the recorded execution
+directly:
+
+```bash
+eigenpal workflow execution list <wf-id> --status failed --limit 5
+eigenpal workflow execution get exec_…
+```
+
+## 2. Pull the full execution
+
+```bash
+eigenpal workflow execution get <exec-id>
+```
+
+The payload includes:
+
+- `status` — `pending` | `running` | `completed` | `failed` | `cancelled` | `skipped`
+- `output` — the workflow's final return value (or partial when failed)
+- `triggerInput` — the input the workflow was called with
+- `stepExecutions` — one entry per step with:
+  - `stepName` — handle from `workflow.yaml`
+  - `status` — same set as the top-level. `skipped` means the step's
+    `if:` evaluated false, OR a `control.foreach` was passed an empty
+    array. Check `skippedReason` for the human-readable cause.
+  - `skippedReason` — populated when `status: 'skipped'`
+  - `input` — fully resolved values (templates already expanded)
+  - `resolvedConfig` — the step's `with:` block AFTER `{{template}}`
+    substitution. Lets you see what the LLM actually got, what URL
+    HTTP actually hit, what items the script actually saw. Populated
+    for `ai.*` / `transform.*` / `action.*`. Null for `control.*`
+    (orchestration steps don't have a meaningful resolved config).
+  - `output` — what the step returned
+  - `error` — error message + code when `status: 'failed'`
+  - `durationMs`
+  - `overrideMode` — `'skipped'` when the example's `meta.json`
+    overrode this step; null otherwise
+
+Quick triage with `jq`:
+
+```bash
+eigenpal workflow execution get <exec-id> \
+  | jq '.stepExecutions[] | { step: .stepName, status, error }'
+```
+
+Narrow to a single step:
+
+```bash
+eigenpal workflow execution get <exec-id> --step extract --include input,output,error
+```
+
+Drill into one field:
+
+```bash
+eigenpal workflow execution get <exec-id> --json | jq '.stepExecutions[2].output.totalAmount'
+```
+
+## 3. Common failure modes
+
+### `validation_failed` on a step input
+
+A step received an input that didn't match its `inputSchema`. Usually
+upstream returned a different shape than expected. Inspect the upstream
+step's `output` and adjust the template expression or schema.
+
+```bash
+# Check the step type's expected input shape
+eigenpal workflow step-type get ai.extract | jq '.configSchema'
+```
+
+### `template_resolution_failed`
+
+A `{{ ... }}` expression referenced a path that didn't exist
+(`steps.foo.output.bar` when `steps.foo.output` is `null`). Either
+guard with `| default:` or fix the upstream step.
+
+### `script_timeout` / `script_memory_limit`
+
+`transform.script` tripped its sandbox limits (5 s wall clock, 10 MB
+heap). Optimize the code or split into smaller steps. The sandbox has
+no network or filesystem, so most timeouts are infinite-loop bugs.
+
+### `step_type_unknown`
+
+`workflow.yaml` references a `type:` that the deployment doesn't know
+about. Common after a typo or a step type that was renamed.
+
+```bash
+eigenpal workflow step-type list --search <partial-match>
+```
+
+### Eval flakiness with `llm-judge`
+
+LLM-judge scores have variance. Stabilize: pin a specific model id
+(`gpt-4o-2024-08-06`), set `temperature: 0`, narrow the prompt to one
+dimension per judge, run multiple experiments and average.
+
+### Workflow `Validation failed` on push
+
+The error envelope tells you exactly where:
+
+```
+✗ Validation failed
+  steps.2.config.passThreshold  Required, got undefined
+  steps.2.type                  Invalid step type: 'eval.llm-judg'
+
+ℹ Failing step is type `eval.llm-judge`. Run
+  `eigenpal workflow step-type get eval.llm-judge` to inspect its
+  config schema.
+```
+
+Follow the hint — every field path is dotted into the YAML so you can
+jump straight to the broken line.
+
+## 4. Compare two executions
+
+```bash
+eigenpal workflow execution compare <exec-a> <exec-b>
+```
+
+Output: per-step status / Δ duration / output diff. Useful for spotting
+regressions between revisions of a workflow.
+
+For a freer-form diff, dump both and compare with `jq`:
+
+```bash
+eigenpal workflow execution get <exec-a> > a.json
+eigenpal workflow execution get <exec-b> > b.json
+diff <(jq -S . a.json) <(jq -S . b.json)
+```
+
+## 5. Cancel a long-running execution
+
+```bash
+eigenpal workflow execution cancel <exec-id>           # TTY: silent
+eigenpal workflow execution cancel <exec-id> --yes     # CI / pipes
+eigenpal workflow execution cancel <exec-id> --json    # raw server payload
+```
+
+The worker finishes the current step, then stops — the cancellation
+check happens between every step transition, including inside
+`control.foreach` / `control.parallel` / `control.block` bodies.
+Behavior by current state:
+
+- `created` / `pending` — transitions straight to `cancelled`.
+- `running` / `waiting` — stamps `cancellationRequestedAt`; the worker
+  observes it on the next step boundary and exits cleanly.
+- already terminal (`completed` / `failed` / `cancelled`) — no-op,
+  exits 0 with an info line. Safe to retry.
+
+## 6. Compare two experiment batches
+
+```bash
+eigenpal workflow experiment compare <batch-a> <batch-b>
+eigenpal workflow experiment compare <batch-a> <batch-b> --regression-threshold 0.10
+eigenpal workflow experiment compare <batch-a> <batch-b> --json | jq '.summary.regressions'
+```
+
+Side-by-side score diff per `(example, evaluator)` pair. Highlights
+regressions vs improvements; aggregate stats at the bottom.
+
+- `--sort abs-delta-desc` (default — biggest movers first) | `delta-asc`
+  (regressions first) | `delta-desc` (improvements first) | `name`
+- `--regression-threshold 0.05` (default) — `Δ < -threshold` is a
+  regression with `⚠`.
+
+**No `--workflow-id` needed.** The server resolves the owning workflow
+from each batch id. Both batches must belong to the same workflow within
+your tenant. (Sibling `experiment` subcommands still take `<wf-id>` as
+the first positional — only `compare` is workflow-agnostic.)
+
+## 7. Connection vs auth errors
+
+`eigenpal status` reports them separately so you can tell which is
+broken. If you see:
+
+```
+✗ Could not connect to https://… — invalid or unreachable base URL.
+  Set EIGENPAL_BASE_URL or pass --base-url <url>.
+```
+
+That's a **connection** error — the URL is wrong, the server is down,
+or your network is blocked. The URL the CLI tried is echoed back so a
+stale `EIGENPAL_BASE_URL` is immediately obvious.
+
+If instead you see:
+
+```
+✗ Not authenticated.
+  Run `eigenpal auth login`, or set EIGENPAL_API_KEY in your env.
+```
+
+That's an **auth** error — the URL works but your key is missing /
+expired / scoped to the wrong tenant. Run `auth list` to see configured
+profiles, or `auth use <name>` to switch.
+
+## 8. Override a flaky step in eval mode
+
+If a step calls an external system that's unreliable (an external API, a
+private API, a rate-limited connector), short-circuit it for evals via
+the example's `meta.json`:
+
+```json
+{
+  "overrides": {
+    "steps": {
+      "<stepName>": { ...stepOutputObject }
+    }
+  }
+}
+```
+
+When the workflow runs against this example, the executor returns the
+override as the step's output without running the real step. The
+execution row shows `overrideMode: 'skipped'` so you can audit which
+steps were faked. Re-import the dataset (`workflow dataset push
+--mode replace`) for the change to land.
+
+## 9. Watching a long batch
+
+```bash
+# Kicks off, prints { batchId, executionIds }
+eigenpal workflow experiment run <wf-id>
+
+# --watch polls until every execution is terminal, prints a per-example
+# failure summary, and exits non-zero if any failed/cancelled.
+eigenpal workflow experiment status <wf-id> --batch-id <id> --watch
+
+# Without --watch: one-shot snapshot, no polling. Prefer --watch in CI / agents.
+eigenpal workflow experiment status <wf-id> --batch-id <id>
+```
+
+The terminal `--watch` output looks like:
+
+```
+7/7 terminal  (failed=7)
+
+✗ 7/7 did not pass
+
+  invoice-2025-001  [failed]  template_resolution_failed at step `parse`: …
+    eigenpal workflow execution get exec_…
+  …
+
+ℹ step-level errors live on `stepExecutions[].error` — see reference/debugging.md.
+```
+
+## 10. Drilling into a single failed execution
+
+The summary surfaces the top-level execution error. For step-by-step
+detail (which step failed, what input it received, what output it
+returned before erroring) follow the suggested command:
+
+```bash
+eigenpal workflow execution get exec_…                                  # full payload
+eigenpal workflow execution get exec_… --step <name> --include input,output,error
+eigenpal workflow execution get exec_… --json | jq -r '.stepExecutions[0].error'
+```
+
+For per-execution live view, `eigenpal workflow execution watch <exec-id>` does
+adaptive polling (2 s while transitioning, 5 s when steady) and prints
+ASCII status badges that work in any terminal.