@eigenpal/cli 0.4.9 → 0.4.11

package/dist/skill/reference/cli/init.md

@@ -26,7 +26,7 @@ init
 | Command                                   | Description                                                                                                                                                             |
 | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `eigenpal init workflow [options] <name>` | Alias of `eigenpal init <name>`. Kept so the `init {workflow,agent}` namespace stays visible — once the agent surface lights up, both kinds will live as siblings here. |
-| `eigenpal init agent [options] <name>`    | Scaffold a new agent (agentic process) project — coming soon. The namespace is reserved so the workflow / agent split is visible from day one.                          |
+| `eigenpal init agent [options] <name>`    | Scaffold a new agent project — coming soon. Use `eigenpal agent push` with an existing agent project for now.                                                           |
 
 ## Details
 
@@ -50,7 +50,7 @@ Alias of `eigenpal init <name>`. Kept so the `init {workflow,agent}` namespace s
 
 ### `eigenpal init agent [options] <name>`
 
-Scaffold a new agent (agentic process) project — coming soon. The namespace is reserved so the workflow / agent split is visible from day one.
+Scaffold a new agent project — coming soon. Use `eigenpal agent push` with an existing agent project for now.
 
 ### Arguments
 

package/dist/skill/reference/evaluators.md

@@ -11,16 +11,41 @@ this is the agent-facing summary.
 ```yaml
 evaluators: # array of evaluator definitions, ≥1 entry
   - name: ... # required, unique within this file, [a-z0-9][a-z0-9-_]*
+    description: ... # optional, ≤500 chars; one-sentence business explanation (see below)
     type: ... # required, one of: exact-diff | llm-judge | custom-script
     config: # required, shape depends on `type`
       passThreshold: 0.95 # 0..1, optional (default 1.0)
       ...
 
-# OPTIONAL global config
-concurrency: 4 # max examples in parallel during `experiment run` (default 1)
-judge: # optional shared model defaults for llm-judge evaluators
-  model: gpt-4o
-  temperature: 0
+# OPTIONAL workflow-level pass threshold. A run passes when its weighted-mean
+# score across evaluators clears this number. Defaults to 0.7.
+passThreshold: 0.7
+```
+
+## Writing descriptions for stakeholders
+
+Every evaluator entry takes an optional `description`. Always set it. The
+dashboard surfaces this string to non-technical reviewers (legal, ops,
+finance) who never see the YAML or the judge prompt. Write it for them:
+
+- One short sentence, plain language.
+- Name what the evaluator measures and why a business reader should care.
+- Skip implementation detail (model names, thresholds, dot-paths). Those
+  belong in `config`.
+- Keep it under ~120 characters when you can; the schema caps it at 500.
+
+```yaml
+# Good. A non-technical reviewer immediately grasps the intent.
+- name: covenants-recall
+  description: Checks every covenant from the contract appears in the extraction.
+  type: llm-judge
+  config: ...
+
+# Bad. Restates the config; teaches a stakeholder nothing.
+- name: covenants-recall
+  description: llm-judge with passThreshold 0.85 over the covenants array.
+  type: llm-judge
+  config: ...
 ```
 
 ## Evaluator types
@@ -47,23 +72,34 @@ judge: # optional shared model defaults for llm-judge evaluators
 - name: covenants-recall
   type: llm-judge
   config:
-    model: gpt-4o # string — the OpenAI model id
-    temperature: 0 # number — for reproducibility
+    model: gpt-4o # OPTIONAL — falls back to the workspace default LLM provider
+    mode: continuous # continuous | discrete (default: continuous)
     passThreshold: 0.85
-    prompt: |
+    docLabel: extraction # what the judge calls the workflow output (default: "output")
+    promptExtension: |
       Score how well the actual output captures every covenant in the
-      expected output, on a scale of 0..1. Penalise missing covenants;
-      do not penalise over-extraction. Return JSON { score, rationale }.
-```
+      expected output. Penalise missing covenants; do not penalise
+      over-extraction. Be strict on covenants that change loan economics.
 
-The harness wraps your `prompt` and feeds:
-
-- `actual` — the workflow's output
-- `expected` — the example's `expected/output.json`
-- `meta` — example metadata
+# discrete mode — the judge picks a label and the score table maps to a number
+- name: tone-check
+  type: llm-judge
+  config:
+    mode: discrete
+    passThreshold: 0.7
+    labels:
+      excellent: 1.0
+      acceptable: 0.7
+      off-tone: 0.0
+    promptExtension: |
+      Pick the label that best matches the tone of the response.
+```
 
-It expects the model to return JSON with at least `score` (0..1) and
-`rationale` (string). The fixed harness handles parse failures + retries.
+`promptExtension` is the only required field — it's appended to the harness
+prompt and tells the judge what "good" looks like for this workflow. The
+harness handles JSON parsing, retries, and feeds the judge `actual`,
+`expected`, and example metadata. In `discrete` mode the judge MUST return
+one of the keys in `labels`; the harness then looks up the score.
 
 ### `custom-script` — JavaScript in the sandbox
 
@@ -72,16 +108,38 @@ It expects the model to return JSON with at least `score` (0..1) and
   type: custom-script
   config:
     passThreshold: 1.0
-    code: |
-      const a = actual.totalAmount ?? 0;
-      const e = expected.totalAmount ?? 0;
-      const score = Math.abs(a - e) <= 0.01 ? 1 : 0;
-      return { score, rationale: `actual=${a}, expected=${e}` };
+    function: |
+      /**
+       * @param {Expected} expected the example's expected output
+       * @param {Actual} actual the workflow's actual output
+       * @returns {number} a score in [0, 1]; throws are caught and scored as 0
+       */
+      function scoreScript(expected, actual) {
+        const a = actual.totalAmount ?? 0;
+        const e = expected.totalAmount ?? 0;
+        return Math.abs(a - e) <= 0.01 ? 1 : 0;
+      }
 ```
 
 Same WASM sandbox as `transform.script` (5s wall clock, 10 MB heap, no
-network). Receives `actual`, `expected`, `meta` in scope. Returns
-`{ score: number, label?: string, rationale?: string }`.
+network). The YAML stores the **whole** `function scoreScript(expected,
+actual) { ... }` declaration verbatim — same text the dashboard editor
+shows. The JSDoc preamble is optional but recommended: it pulls the
+workflow's output schema into IntelliSense for `expected.…` /
+`actual.…` in the dashboard.
+
+The signature shape is **locked**:
+
+- Function name MUST be `scoreScript`.
+- Parameters MUST be `(expected, actual)` in that order. Swapping them
+  would silently invert every score.
+- Body MUST `return` a number in `[0, 1]`, or `throw` (which is caught
+  and scored as 0).
+
+`workflow validate` and `workflow evaluators validate` flag YAML where
+the function fails to parse, the signature doesn't match, or the body
+contains neither `return` nor `throw` — so a hand-edited config errors
+before it reaches the worker.
 
 ## Pass / fail vs score
 
@@ -160,9 +218,11 @@ process — `--out` writes the file directly from the server's storage.
 LLM-judge scores have variance. To reduce flake:
 
 - Pin a specific model id (`gpt-4o-2024-08-06` rather than `gpt-4o`).
-- Set `temperature: 0`.
-- Keep the prompt narrow — score one dimension per evaluator. Use
+- Keep `promptExtension` narrow — score one dimension per evaluator. Use
   multiple judges instead of one omnibus prompt.
+- Prefer `mode: discrete` with a small label set when the judgement is
+  categorical (good / acceptable / bad) — it eliminates the model's
+  free-form numeric drift.
 - For high-stakes thresholds, run the experiment ≥3 times and average.
 
 <!-- GENERATED:EVALUATOR_CATALOG START -->
@@ -177,6 +237,14 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | `evaluators` | array<object> | no | `[]` |  |
 | `passThreshold` | number | no | `0.7` |  |
 
+### Common entry fields (every evaluator type)
+
+| Field | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `name` | string | yes |  | Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall. |
+| `description` | string | no |  | One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`. |
+| `weight` | number | no | `1` | Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall. |
+
 ### `exact-diff` — JSON deep-diff against expected output
 
 | Field | Type | Required | Default | Description |
@@ -203,7 +271,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
-| `code` | string | yes |  | Sandboxed JavaScript. Receives `actual`, `expected`, `input` as variables and must return { score, label? } where score is in [0, 1]. |
+| `function` | string | yes |  | Full JavaScript declaration of `function scoreScript(expected, actual) { ... }`. Receives `expected` (the example's expected output) and `actual` (the workflow's actual output), returns a number in [0, 1]. Throws are caught and scored as 0. |
 | `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it's killed and the run is marked failed. |
 | `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
 | `passThreshold` | number | no | `1` | Minimum script score this evaluator must reach for a run to pass. |

package/dist/skill/reference/step-types.md

@@ -70,21 +70,10 @@ the catalog tells you what fields it takes.
 - Real JS — `Array.prototype.reduce`, `JSON.parse`, etc.
 - Sandbox limits prevent infinite loops from blowing up an execution
 
-> **Common gotcha — TDZ on shadowed input names.** Each entry in `inputs`
-> becomes a top-level `const` binding inside your script. Re-declaring the
-> same name in the body shadows the binding and triggers a `ReferenceError:
-> Cannot access 'X' before initialization` (Temporal Dead Zone). For example:
->
-> ```js
-> // ✗ TDZ — `located` is the input AND the inner const
-> const located = located || [];
->
-> // ✓ Rename the inner variable
-> const safeLocated = located ?? [];
-> ```
->
-> If you need a default, use `??` / `||` against the binding directly without
-> redeclaring, or rename the local. The same applies to `let` redeclarations.
+> Each entry in `inputs:` becomes a parameter of `function script(...)`
+> in declaration order. `inputs: { items, taxRate }` ⇒
+> `function script(items, taxRate) { … }`. The function must `return`
+> (or `throw`) a value.
 
 ## Reference / output paths
 
@@ -329,14 +318,14 @@ Convert XLSX spreadsheet to JSON array of row objects for use in scripts or down
 
 #### `transform.script` — Script
 
-Execute JavaScript code in a secure sandbox. Input keys become TOP-LEVEL variables (use "items" not "inputs.items"). Must return a value.
+Execute a JavaScript function in a QuickJS sandbox. Input keys become the function's parameter list, in declaration order: `inputs: { items, taxRate }` ⇒ `function script(items, taxRate) { … }`. Must `return` (or `throw`) a value.
 
 **Config** (in `step.with`):
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
-| `inputs` | record<string, string> | no |  | Named inputs mapped from template expressions. IMPORTANT: Keys become TOP-LEVEL variables in code (e.g., if key is "items", use "items" not "inputs.items"). |
-| `code` | string | yes |  | JavaScript code. Input keys are available as top-level variables (NOT as inputs.key). Must return a value. |
+| `inputs` | record<string, string> | no |  | Named inputs mapped from template expressions. Keys become the function parameter list in declaration order: `inputs: { items, taxRate }` ⇒ `function script(items, taxRate) { … }`. |
+| `function` | string | no |  | JavaScript function declaration. Must be `function script(arg1, arg2, …) { … }` where the parameter list equals `Object.keys(inputs)` in declaration order. Must `return` (or `throw`) a value. |
 | `outputSchema` | record<string, unknown> | no |  | JSON Schema describing the expected return value. Used for validation and type hints. |
 | `timeout` | number | no | `5000` | Max execution time in milliseconds (default: 5000) |
 | `memoryLimit` | number | no | `10485760` | Max memory in bytes (default: 10MB) |

package/dist/skill/reference/workflow-yaml.md

@@ -158,10 +158,12 @@ arithmetic, regex) goes in `transform.script`:
     inputs: # explicit data deps — visible to the validator
       lineItems: '{{ steps.extract.output.lineItems }}'
       taxRate: '{{ input.taxRate }}'
-    code: |
-      const subtotal = lineItems.reduce((s, i) => s + i.price * i.qty, 0);
-      const tax = subtotal * taxRate;
-      return { subtotal, tax, total: subtotal + tax };
+    function: |
+      function script(lineItems, taxRate) {
+        const subtotal = lineItems.reduce((s, i) => s + i.price * i.qty, 0);
+        const tax = subtotal * taxRate;
+        return { subtotal, tax, total: subtotal + tax };
+      }
     outputSchema: # optional but recommended — runtime-validated
       type: object
       properties:

package/dist/templates/blank/workflow.yaml

@@ -25,8 +25,10 @@ steps:
     with:
       inputs:
         text: '{{ input.text }}'
-      code: |
-        return { length: text.length, original: text };
+      function: |
+        function script(text) {
+          return { length: text.length, original: text };
+        }
       outputSchema:
         type: object
         properties:

package/dist/templates/pdf-extraction/evaluators.yaml

@@ -1,5 +1,6 @@
 evaluators:
   - name: exact-match
+    description: Checks the extracted fields match the expected output exactly.
     type: exact-diff
     config:
       passThreshold: 1.0

package/dist/templates/text-classification/evaluators.yaml

@@ -1,5 +1,6 @@
 evaluators:
   - name: label-match
+    description: Checks the predicted label matches the expected label.
     type: exact-diff
     config:
       passThreshold: 1.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eigenpal/cli",
-  "version": "0.4.9",
+  "version": "0.4.11",
   "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   "keywords": [
     "eigenpal",