@eigenpal/cli 0.0.1 → 0.4.1

package/dist/skill/reference/evaluators.md

@@ -0,0 +1,210 @@
+# evaluators.yaml — required schema
+
+Evaluators run after a workflow execution to grade its output. They
+attach to a workflow via `workflow evaluators push` and run in batch via
+`workflow experiment run`. The full Zod schema is
+`EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-config.ts`;
+this is the agent-facing summary.
+
+## Top-level shape
+
+```yaml
+evaluators: # array of evaluator definitions, ≥1 entry
+  - name: ... # required, unique within this file, [a-z0-9][a-z0-9-_]*
+    type: ... # required, one of: exact-diff | llm-judge | custom-script
+    config: # required, shape depends on `type`
+      passThreshold: 0.95 # 0..1, optional (default 1.0)
+      ...
+
+# OPTIONAL global config
+concurrency: 4 # max examples in parallel during `experiment run` (default 1)
+judge: # optional shared model defaults for llm-judge evaluators
+  model: gpt-4o
+  temperature: 0
+```
+
+## Evaluator types
+
+### `exact-diff` — JSON deep-diff against `expected/output.json`
+
+```yaml
+- name: invoice-fields
+  type: exact-diff
+  config:
+    passThreshold: 1.0 # 1.0 = byte-equal; lower = allow some slack
+    paths: [invoiceNumber, totalAmount] # OPTIONAL — restrict to specific dotted paths
+```
+
+- `passThreshold` — number in `[0, 1]`. The diff produces a similarity
+  score; `score >= passThreshold` ⇒ `passed: true`.
+- `paths` — array of dotted paths into the actual / expected outputs.
+  When set, only those paths are diffed. When absent, the whole object
+  is compared.
+
+### `llm-judge` — LLM-as-judge scoring
+
+```yaml
+- name: covenants-recall
+  type: llm-judge
+  config:
+    model: gpt-4o # string — the OpenAI model id
+    temperature: 0 # number — for reproducibility
+    passThreshold: 0.85
+    prompt: |
+      Score how well the actual output captures every covenant in the
+      expected output, on a scale of 0..1. Penalise missing covenants;
+      do not penalise over-extraction. Return JSON { score, rationale }.
+```
+
+The harness wraps your `prompt` and feeds:
+
+- `actual` — the workflow's output
+- `expected` — the example's `expected/output.json`
+- `meta` — example metadata
+
+It expects the model to return JSON with at least `score` (0..1) and
+`rationale` (string). The fixed harness handles parse failures + retries.
+
+### `custom-script` — JavaScript in the sandbox
+
+```yaml
+- name: total-with-tolerance
+  type: custom-script
+  config:
+    passThreshold: 1.0
+    code: |
+      const a = actual.totalAmount ?? 0;
+      const e = expected.totalAmount ?? 0;
+      const score = Math.abs(a - e) <= 0.01 ? 1 : 0;
+      return { score, rationale: `actual=${a}, expected=${e}` };
+```
+
+Same WASM sandbox as `transform.script` (5s wall clock, 10 MB heap, no
+network). Receives `actual`, `expected`, `meta` in scope. Returns
+`{ score: number, label?: string, rationale?: string }`.
+
+## Pass / fail vs score
+
+Every evaluator emits both:
+
+- `score` — numeric, typically `[0, 1]`
+- `passed` — boolean, derived from `score >= passThreshold`
+
+The batch pass-rate counts examples where **every** evaluator on the
+workflow passed. A failing single evaluator is enough to fail the
+example overall.
+
+## Per-example overrides
+
+Most useful for skipping steps that need external state. Example's
+`meta.json`:
+
+```json
+{
+  "overrides": {
+    "steps": {
+      "enrich_company": {
+        "companyName": "Acme Corp",
+        "registeredAt": "2020-01-01"
+      }
+    }
+  }
+}
+```
+
+When the workflow runs against this example, `enrich_company` is
+short-circuited and the override JSON is used as the step's output.
+The execution row records `overrideMode: 'skipped'` so you can audit
+which steps were faked. See `dataset-format.md` for the full meta.json
+schema.
+
+## Validate before pushing
+
+```bash
+eigenpal workflow evaluators validate ./evaluators.yaml
+```
+
+Reports field-level issues:
+
+```
+✗ evaluators (./evaluators.yaml) — 1 issue
+  evaluators.0.config.passThreshold  Number must be less than or equal to 1
+```
+
+## Push + run
+
+```bash
+# Push the evaluators (overwrites the workflow's existing config).
+eigenpal workflow evaluators push <wf-id> --file evaluators.yaml
+
+# Kick off a server-side batch eval over the whole pushed dataset.
+# Returns { batchId, executionIds }.
+eigenpal workflow experiment run <wf-id>
+
+# Or restrict to a single example:
+eigenpal workflow experiment run <wf-id> --example-id evx_…
+
+# Watch progress.
+eigenpal workflow experiment status <wf-id> --batch-id <id>
+
+# Pull results when done.
+eigenpal workflow experiment results <wf-id> --batch-id <id> \
+  --format csv --out results.csv
+```
+
+The signed-URL pattern means large result sets don't go through the CLI
+process — `--out` writes the file directly from the server's storage.
+
+## Stabilizing flaky judges
+
+LLM-judge scores have variance. To reduce flake:
+
+- Pin a specific model id (`gpt-4o-2024-08-06` rather than `gpt-4o`).
+- Set `temperature: 0`.
+- Keep the prompt narrow — score one dimension per evaluator. Use
+  multiple judges instead of one omnibus prompt.
+- For high-stakes thresholds, run the experiment ≥3 times and average.
+
+<!-- GENERATED:EVALUATOR_CATALOG START -->
+## Evaluator types — full reference
+
+_Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-config.ts`. Do not hand-edit between the GENERATED fences — run `bun run --cwd packages/cli generate:skill`._
+
+### Top-level `evaluators.yaml`
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `evaluators` | array<object> | no | `[]` |  |
+| `passThreshold` | number | no | `0.7` |  |
+
+### `exact-diff` — JSON deep-diff against expected output
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `numericTolerance` | number | no | `0.000001` | Maximum absolute difference between numeric values that still counts as equal. Use 1e-6 for floats, 0 for exact integer match. |
+| `allowExtraFields` | boolean | no | `true` | When on, extra fields in the actual output don't fail the diff. Off = actual must match expected exactly. |
+| `passThreshold` | number | no | `1` | Minimum diff score this evaluator must reach for a run to pass. 1.0 = perfect match required. |
+| `paths` | array<string> | no |  | Dot-paths to scope the diff to a subset of the expected tree. Empty = diff entire expected. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`. Use this when expected has noisy sections you don't want to score. |
+
+
+### `llm-judge` — LLM-as-judge scoring
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `model` | string | no |  | Which LLM grades the output. Falls back to the workspace's default LLM provider when unset. |
+| `mode` | `"continuous"` \| `"discrete"` | no | `"continuous"` | Continuous = the LLM returns a free-form score in [0, 1]. Discrete = the LLM picks one of your labels and the score is looked up from the table. |
+| `labels` | record<string, number> | no |  | Allowed labels and the score each maps to. Required in discrete mode. The judge MUST return one of these labels; each score must be in [0, 1]. |
+| `passThreshold` | number | no | `0.7` | Minimum judge score this evaluator must reach for a run to pass. |
+| `promptExtension` | string | yes |  | Required. Evaluation criteria appended to the judge prompt — describe what makes a good answer for this workflow (tone, completeness, edge cases). Cannot be empty: an empty prompt yields the model inventing its own criteria. |
+| `docLabel` | string | no | `"output"` | Name the judge uses to refer to the output in its prompt (e.g. 'invoice', 'extraction'). Default is 'output'. |
+
+
+### `custom-script` — JavaScript in the sandbox
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `code` | string | yes |  | Sandboxed JavaScript. Receives `actual`, `expected`, `input` as variables and must return { score, label? } where score is in [0, 1]. |
+| `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it's killed and the run is marked failed. |
+| `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
+| `passThreshold` | number | no | `1` | Minimum script score this evaluator must reach for a run to pass. |
+<!-- GENERATED:EVALUATOR_CATALOG END -->

package/dist/skill/reference/step-exec.md

@@ -0,0 +1,164 @@
+# Run one step locally — `workflow step exec`
+
+Tightest iteration loop in the CLI. One step, no server, no queue, no
+DAG. Milliseconds per run. Use this when you're tweaking a single
+step's logic; use `workflow execution run` when you need the whole
+workflow.
+
+```bash
+eigenpal workflow step exec <type> [flags]
+```
+
+`<type>` is any value from `eigenpal workflow step-type list`. Only
+some types have a local runner today (table below). The rest exit 2
+with a "not yet supported locally" message — they still need the
+server.
+
+## What runs locally today
+
+| Step type | Runner | Notes |
+| --- | --- | --- |
+| `transform.script` | QuickJS sandbox in-process | Same sandbox as prod. TDZ traps and timeout / memory limits reproduce here. |
+| `ai.extract` | OpenAI SDK | Reads `WORKER_LLM_API_KEY` (or `OPENAI_API_KEY` for openai/openai-compatible providers) from env. Real network call. |
+| anything else | — | Exits 2 with "isn't yet supported by 'workflow step exec' locally". |
+
+If the runner you need isn't here yet, fall back to `workflow execution run`
+on the server (slower loop, but covers every step type).
+
+## Inputs — `--inputs k=v`
+
+Repeatable. Three value forms:
+
+```bash
+--inputs limit=10                # inline literal (string; numeric strings stay strings)
+--inputs items=@items.json       # file contents (JSON-parsed if valid; raw text otherwise)
+--inputs payload=@-              # stdin (the `@` prefix is required; bare `-` is a literal)
+```
+
+For `transform.script`, every `--inputs k=v` becomes a top-level
+binding in the sandbox (so `k` is referenced as `k` inside the code,
+mirroring runtime semantics — including TDZ traps for `const x = x || []`).
+
+For `ai.extract`, `--inputs input=…` is the document text.
+
+## Config — `--config-json` or `--config-file`
+
+Mutually exclusive. The shape is whatever the step type's `with:` block
+accepts (see `eigenpal workflow step-type get <type>`).
+
+```bash
+--config-json '{"code":"return items.reduce((s,i)=>s+i.v,0)"}'
+--config-file ./step-config.json
+--config-file -                  # config from stdin (e.g. piped from `workflow pull`)
+```
+
+## Output
+
+Goes to **stdout as JSON**. Status messages go to stderr — pipe stdout
+into `jq` cleanly:
+
+```bash
+eigenpal workflow step exec transform.script --config-file cfg.json --inputs items=@arr.json | jq '.'
+```
+
+## Output schema validation — `--output-schema`
+
+Optional. Pass a JSON Schema file:
+
+```bash
+--output-schema ./expected-schema.json
+```
+
+When omitted, the CLI falls back to the step type's built-in
+`outputSchema` from `STEP_SCHEMAS` — so even without the flag, output
+shape gets checked.
+
+A violation **exits 2** with a machine-readable envelope on stderr:
+
+```json
+{ "code": "output_schema_violation", "issues": [
+  { "field": "/total", "message": "must be number" }
+]}
+```
+
+`field` is the JSON Pointer to the offending value (Ajv's `instancePath`,
+remapped to `field` for stability) or `<root>` if the whole output is
+wrong. Agents can parse this directly — no text scraping.
+
+## Resource limits (transform.script only)
+
+```bash
+--timeout-ms 10000      # default 5000 (5 s wall clock)
+--memory-mb 25          # default 10 MB heap
+```
+
+Use these when iterating on a memory-heavy reducer. They override the
+config's `timeout` / `memoryLimit` fields just for this run.
+
+## Worked examples
+
+### Tweak a JS reducer until tests pass
+
+Put the config in a file so the JS doesn't need escaping:
+
+```bash
+cat > cfg.json <<'EOF'
+{ "code": "return items.reduce((sum, i) => sum + i.value, 0)" }
+EOF
+echo '[{"value":1},{"value":2},{"value":3}]' > items.json
+
+eigenpal workflow step exec transform.script --config-file cfg.json --inputs items=@items.json
+# 6
+```
+
+For a one-liner without a config file, `--config-json '{"code":"..."}'`
+works too — but mind the shell escaping.
+
+### Run an extract once with sample text
+
+```bash
+cat > extract.json <<'EOF'
+{
+  "prompt": "Extract the invoice total as a number.",
+  "schema": { "type": "object", "properties": { "total": { "type": "number" } } }
+}
+EOF
+
+eigenpal workflow step exec ai.extract \
+  --config-file extract.json \
+  --inputs input=@invoice.txt
+# {"total": 1234.56}
+```
+
+### Pipe config straight from a workflow YAML
+
+When you've already pushed a workflow and want to re-run one of its
+steps locally:
+
+```bash
+eigenpal workflow pull wf_abc > workflow.yaml
+jq '.steps[] | select(.name == "extract") | .with' workflow.yaml \
+  | eigenpal workflow step exec ai.extract --config-file - --inputs input=@invoice.txt
+```
+
+## Exit codes
+
+| Code | When |
+| --- | --- |
+| 0 | Step ran; output validated. |
+| 1 | Runtime crash (sandbox blew up; `ai.extract` network failure; unexpected exception). |
+| 2 | Misuse OR `output_schema_violation` (envelope on stderr) OR unknown / unsupported step type. |
+
+Exit 2 is always recoverable — fix your input / config / schema and
+re-run. Exit 1 means something is broken upstream (network, runtime,
+your code).
+
+## Discovery
+
+```bash
+eigenpal workflow step-type list              # all registered types
+eigenpal workflow step-type get <type>        # config + output schema
+```
+
+The catalog is generated from `STEP_SCHEMAS` in `@eigenpal/types` and
+is always in sync with what the deployment supports.