@eigenpal/cli 0.14.2 → 0.15.0

package/dist/skill/reference/evaluators.md

@@ -77,6 +77,8 @@ inherited settings. `{}` selects a path with inherited platform defaults.
         allowExtraFields: false
       lineItems[].unitPrice:
         numericTolerance: 0.01
+      lineItems[].evidence:
+        ignore: true
 ```
 
 - `rules` — map of path → comparison settings. Only listed paths (and their
@@ -114,6 +116,10 @@ inherited settings. `{}` selects a path with inherited platform defaults.
   `lineItems`, not `lineItems[]`.
 - `allowExtraFields: false` — on an object path (often `someArray[]`), reject
   object keys present in actual but absent from expected.
+- `ignore: true` — remove that path and every descendant from both expected and
+  actual output. This is useful for volatile evidence, citations, and timestamps
+  under an otherwise selected parent. It cannot share a rule with other
+  settings. An ignore-only rules map means full output minus those paths.
 
 Path syntax: `lineItems` targets the array; `lineItems[]` targets each element;
 `lineItems[].unitPrice` targets a field inside every element. Nested unordered
@@ -308,10 +314,10 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
-| `evaluators` | array<object> | no | `[]` |  |
-| `passThreshold` | number | no |  |  |
+| `evaluators` | array<object> | no | `[]` | Evaluator entries run after each experiment execution. |
+| `passThreshold` | number | no |  | Workflow-level pass threshold for the weighted mean score. When omitted, legacy evaluator thresholds are combined for compatibility, otherwise the platform default is 0.7. |
 
-### Common entry fields (every evaluator type)
+### Evaluator entry envelope
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
@@ -319,11 +325,14 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | `description` | string | no |  | One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`. |
 | `weight` | number | no | `1` | Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall. |
 
+- `type` — required discriminator: `exact-diff`, `llm-judge`, `custom-script`.
+- `config` — required object whose fields are defined by the selected evaluator type below.
+
 ### `exact-diff` — JSON deep-diff against expected output
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
-| `rules` | record<string, object> | no |  | Per-path comparison rules. Keys select compared paths (`$` = full output). More-specific keys inherit then override parent rules. `{}` selects a path using inherited platform defaults. Do not mix with legacy `paths`, `unorderedPaths`, `numericTolerance`, or `allowExtraFields`. |
+| `rules` | record<string, object> | no |  | Per-path comparison rules. Non-ignored keys select compared paths (`$` = full output); an ignore-only map means full output minus those paths. More-specific keys inherit then override parent rules. `{}` selects a path using inherited platform defaults; `{ ignore: true }` excludes it. Do not mix with legacy `paths`, `unorderedPaths`, `numericTolerance`, or `allowExtraFields`. |
 | `numericTolerance` | number | no |  | Legacy global numeric tolerance (absolute). When omitted, comparison uses historical relative epsilon. Prefer `rules.<path>.numericTolerance`. Accepted only when `rules` is omitted. |
 | `allowExtraFields` | boolean | no |  | Legacy global extra-field/extra-item flag. Prefer `rules.<path>.allowExtraFields` and `rules.<path>.items`. `true` normalizes to object extras allowed and array `items: at-least`; `false` normalizes to object extras rejected and array `items: exactly`. Accepted only when `rules` is omitted. |
 | `passThreshold` | number | no |  | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
@@ -334,6 +343,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
+| `ignore` | `true` | no |  | Exclude this path and every descendant from comparison. Cannot be combined with other settings on the same rule. |
 | `order` | `"ordered"` \| `"unordered"` | no |  | Array matching mode at this path. Independent of `items`. Set on the array path (`lineItems`), not the item path (`lineItems[]`). `unordered` matches distinct items in any order. Arrays stay ordered when omitted. Unordered matching is limited to 1000 items, 1000000 candidate pairs, and 2000000 recursive operations per comparison. |
 | `items` | `"at-least"` \| `"at-most"` \| `"exactly"` | no |  | How expected and actual array items relate. Set on the array path (`lineItems`), not the item path (`lineItems[]`). Independent of `order`. Ordered arrays use positional prefix matching: `at-least` (default) requires expected to match actual from index 0 and allows trailing actual extras; `at-most` requires actual to match expected from index 0 and allows trailing expected extras; `exactly` is equal-cardinality positional comparison. Unordered arrays match distinct items in any order: `at-least` requires every expected item and allows extra actual items (including extras with missing or duplicate identities); `at-most` requires every actual item and allows missing expected items (every actual item being checked needs a valid unique identity); `exactly` is a bijection (both sides need valid unique identities). |
 | `matchBy` | string \| array<string> | no |  | Identity fields for unordered arrays of objects. Requires `order: unordered` on the same rule. Set on the array path (`lineItems`), not the item path (`lineItems[]`). A relative path (`sku`) or unique paths (`[country, sku]`) inside each item. Pairs expected and actual objects by exact identity, then diffs the paired objects with inherited nested rules. Identity type and value must both match: numeric `10` differs from string `'10'`; no coercion or numeric tolerance. Omitted = structural matching. Applies only at the array path where it is set and is not inherited by descendants. Nested object fields are allowed; array indexes, commas, and wildcards are not. Empty or comma-only values are rejected; composite paths must be unique. |
@@ -346,6 +356,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
 | `model` | string | no |  | Which LLM grades the output. Falls back to the workspace's default LLM provider when unset. |
+| `reasoningEffort` | `"none"` \| `"minimal"` \| `"low"` \| `"medium"` \| `"high"` \| `"xhigh"` \| `"max"` | no |  | Reasoning effort for models that support it. Omit to preserve the current provider default. |
 | `mode` | `"continuous"` \| `"discrete"` | no | `"continuous"` | Continuous = the LLM returns a free-form score in [0, 1]. Discrete = the LLM picks one of your labels and the score is looked up from the table. |
 | `labels` | record<string, number> | no |  | Allowed labels and the score each maps to. Required in discrete mode. The judge MUST return one of these labels; each score must be in [0, 1]. |
 | `passThreshold` | number | no |  | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
@@ -361,4 +372,351 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | `timeoutMs` | integer | no | `5000` | Maximum wall-clock time the script can run before it is killed and the run is marked failed. |
 | `memoryLimitMb` | integer | no | `10` | Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings. |
 | `passThreshold` | number | no |  | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
+
+
+### Complete machine-readable `evaluators.yaml` schema
+
+Generated from the same discriminated union used by evaluator validation; YAML keys and nesting are identical.
+
+```json
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "properties": {
+    "evaluators": {
+      "default": [],
+      "description": "Evaluator entries run after each experiment execution.",
+      "type": "array",
+      "items": {
+        "oneOf": [
+          {
+            "type": "object",
+            "properties": {
+              "name": {
+                "type": "string",
+                "minLength": 1,
+                "maxLength": 64,
+                "pattern": "^[a-z0-9-]+$",
+                "description": "Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall."
+              },
+              "description": {
+                "description": "One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`.",
+                "type": "string",
+                "maxLength": 500
+              },
+              "weight": {
+                "default": 1,
+                "description": "Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall.",
+                "type": "number",
+                "minimum": 0
+              },
+              "type": {
+                "type": "string",
+                "const": "exact-diff",
+                "description": "Deterministic structured-output comparison."
+              },
+              "config": {
+                "default": {},
+                "description": "Configuration for exact-diff.",
+                "type": "object",
+                "properties": {
+                  "rules": {
+                    "description": "Per-path comparison rules. Non-ignored keys select compared paths (`$` = full output); an ignore-only map means full output minus those paths. More-specific keys inherit then override parent rules. `{}` selects a path using inherited platform defaults; `{ ignore: true }` excludes it. Do not mix with legacy `paths`, `unorderedPaths`, `numericTolerance`, or `allowExtraFields`.",
+                    "type": "object",
+                    "propertyNames": {
+                      "type": "string"
+                    },
+                    "additionalProperties": {
+                      "type": "object",
+                      "properties": {
+                        "ignore": {
+                          "description": "Exclude this path and every descendant from comparison. Cannot be combined with other settings on the same rule.",
+                          "type": "boolean",
+                          "const": true
+                        },
+                        "order": {
+                          "description": "Array matching mode at this path. Independent of `items`. Set on the array path (`lineItems`), not the item path (`lineItems[]`). `unordered` matches distinct items in any order. Arrays stay ordered when omitted. Unordered matching is limited to 1000 items, 1000000 candidate pairs, and 2000000 recursive operations per comparison.",
+                          "type": "string",
+                          "enum": [
+                            "ordered",
+                            "unordered"
+                          ]
+                        },
+                        "items": {
+                          "description": "How expected and actual array items relate. Set on the array path (`lineItems`), not the item path (`lineItems[]`). Independent of `order`. Ordered arrays use positional prefix matching: `at-least` (default) requires expected to match actual from index 0 and allows trailing actual extras; `at-most` requires actual to match expected from index 0 and allows trailing expected extras; `exactly` is equal-cardinality positional comparison. Unordered arrays match distinct items in any order: `at-least` requires every expected item and allows extra actual items (including extras with missing or duplicate identities); `at-most` requires every actual item and allows missing expected items (every actual item being checked needs a valid unique identity); `exactly` is a bijection (both sides need valid unique identities).",
+                          "type": "string",
+                          "enum": [
+                            "at-least",
+                            "at-most",
+                            "exactly"
+                          ]
+                        },
+                        "matchBy": {
+                          "description": "Identity fields for unordered arrays of objects. Requires `order: unordered` on the same rule. Set on the array path (`lineItems`), not the item path (`lineItems[]`). A relative path (`sku`) or unique paths (`[country, sku]`) inside each item. Pairs expected and actual objects by exact identity, then diffs the paired objects with inherited nested rules. Identity type and value must both match: numeric `10` differs from string `'10'`; no coercion or numeric tolerance. Omitted = structural matching. Applies only at the array path where it is set and is not inherited by descendants. Nested object fields are allowed; array indexes, commas, and wildcards are not. Empty or comma-only values are rejected; composite paths must be unique.",
+                          "anyOf": [
+                            {
+                              "type": "string",
+                              "minLength": 1,
+                              "pattern": "^[^.[\\]\\s$,]+(?:\\.[^.[\\]\\s$,]+)*$"
+                            },
+                            {
+                              "minItems": 1,
+                              "type": "array",
+                              "items": {
+                                "type": "string",
+                                "minLength": 1,
+                                "pattern": "^[^.[\\]\\s$,]+(?:\\.[^.[\\]\\s$,]+)*$"
+                              }
+                            }
+                          ]
+                        },
+                        "allowExtraFields": {
+                          "description": "When false, extra actual keys at this object fail the diff. When omitted, extra keys are allowed. Applies only to objects.",
+                          "type": "boolean"
+                        },
+                        "numericTolerance": {
+                          "description": "Maximum absolute difference for numeric leaves at or under this path. Omitted = inherit, then historical relative epsilon. Explicit `0` is exact.",
+                          "type": "number",
+                          "minimum": 0
+                        }
+                      },
+                      "additionalProperties": false
+                    }
+                  },
+                  "numericTolerance": {
+                    "description": "Legacy global numeric tolerance (absolute). When omitted, comparison uses historical relative epsilon. Prefer `rules.<path>.numericTolerance`. Accepted only when `rules` is omitted.",
+                    "type": "number",
+                    "minimum": 0
+                  },
+                  "allowExtraFields": {
+                    "description": "Legacy global extra-field/extra-item flag. Prefer `rules.<path>.allowExtraFields` and `rules.<path>.items`. `true` normalizes to object extras allowed and array `items: at-least`; `false` normalizes to object extras rejected and array `items: exactly`. Accepted only when `rules` is omitted.",
+                    "type": "boolean"
+                  },
+                  "passThreshold": {
+                    "description": "Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model.",
+                    "type": "number",
+                    "minimum": 0,
+                    "maximum": 1
+                  },
+                  "paths": {
+                    "description": "Legacy scoped paths. Prefer selecting those paths as `rules` keys. Syntax: `header.id`, `lineItems[].total`, `lineItems[0].sku`.",
+                    "type": "array",
+                    "items": {
+                      "type": "string",
+                      "minLength": 1
+                    }
+                  },
+                  "unorderedPaths": {
+                    "description": "Legacy unordered array paths. Prefer `rules.<path>.order: unordered`. Name the array field itself (`subjects`, not `subjects[]`).",
+                    "type": "array",
+                    "items": {
+                      "type": "string",
+                      "minLength": 1,
+                      "pattern": "^[^.[\\]\\s]+(?:(?:\\[\\])?\\.[^.[\\]\\s]+)*$"
+                    }
+                  }
+                },
+                "additionalProperties": false
+              }
+            },
+            "required": [
+              "name",
+              "weight",
+              "type",
+              "config"
+            ],
+            "additionalProperties": false
+          },
+          {
+            "type": "object",
+            "properties": {
+              "name": {
+                "type": "string",
+                "minLength": 1,
+                "maxLength": 64,
+                "pattern": "^[a-z0-9-]+$",
+                "description": "Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall."
+              },
+              "description": {
+                "description": "One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`.",
+                "type": "string",
+                "maxLength": 500
+              },
+              "weight": {
+                "default": 1,
+                "description": "Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall.",
+                "type": "number",
+                "minimum": 0
+              },
+              "type": {
+                "type": "string",
+                "const": "llm-judge",
+                "description": "Model-graded semantic comparison."
+              },
+              "config": {
+                "type": "object",
+                "properties": {
+                  "model": {
+                    "description": "Which LLM grades the output. Falls back to the workspace's default LLM provider when unset.",
+                    "type": "string",
+                    "minLength": 1
+                  },
+                  "reasoningEffort": {
+                    "description": "Reasoning effort for models that support it. Omit to preserve the current provider default.",
+                    "type": "string",
+                    "enum": [
+                      "none",
+                      "minimal",
+                      "low",
+                      "medium",
+                      "high",
+                      "xhigh",
+                      "max"
+                    ]
+                  },
+                  "mode": {
+                    "default": "continuous",
+                    "description": "Continuous = the LLM returns a free-form score in [0, 1]. Discrete = the LLM picks one of your labels and the score is looked up from the table.",
+                    "type": "string",
+                    "enum": [
+                      "continuous",
+                      "discrete"
+                    ]
+                  },
+                  "labels": {
+                    "description": "Allowed labels and the score each maps to. Required in discrete mode. The judge MUST return one of these labels; each score must be in [0, 1].",
+                    "type": "object",
+                    "propertyNames": {
+                      "type": "string",
+                      "minLength": 1
+                    },
+                    "additionalProperties": {
+                      "type": "number",
+                      "minimum": 0,
+                      "maximum": 1
+                    }
+                  },
+                  "passThreshold": {
+                    "description": "Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model.",
+                    "type": "number",
+                    "minimum": 0,
+                    "maximum": 1
+                  },
+                  "promptExtension": {
+                    "type": "string",
+                    "minLength": 10,
+                    "description": "Required. Evaluation criteria appended to the judge prompt — describe what makes a good answer for this workflow (tone, completeness, edge cases). Cannot be empty: an empty prompt yields the model inventing its own criteria."
+                  },
+                  "docLabel": {
+                    "default": "output",
+                    "description": "Name the judge uses to refer to the output in its prompt (e.g. 'invoice', 'extraction'). Default is 'output'.",
+                    "type": "string"
+                  }
+                },
+                "required": [
+                  "mode",
+                  "promptExtension",
+                  "docLabel"
+                ],
+                "additionalProperties": false,
+                "description": "Configuration for llm-judge."
+              }
+            },
+            "required": [
+              "name",
+              "weight",
+              "type",
+              "config"
+            ],
+            "additionalProperties": false
+          },
+          {
+            "type": "object",
+            "properties": {
+              "name": {
+                "type": "string",
+                "minLength": 1,
+                "maxLength": 64,
+                "pattern": "^[a-z0-9-]+$",
+                "description": "Identifier used in eval results and the weighted-mean formula. Use a short kebab-case slug like header-exact or covenants-recall."
+              },
+              "description": {
+                "description": "One-sentence explanation, in business language, of what this evaluator measures and why a stakeholder should care. Shown in the dashboard to non-technical reviewers (legal, ops, finance) who never see the YAML or judge prompt. Skip implementation detail (model names, thresholds, dot-paths); those live in `config`.",
+                "type": "string",
+                "maxLength": 500
+              },
+              "weight": {
+                "default": 1,
+                "description": "Relative importance in the weighted mean. The overall score is Σ(weightᵢ × scoreᵢ) / Σweight. Set to 0 to score-only without affecting overall.",
+                "type": "number",
+                "minimum": 0
+              },
+              "type": {
+                "type": "string",
+                "const": "custom-script",
+                "description": "Typed deterministic scoring script."
+              },
+              "config": {
+                "type": "object",
+                "properties": {
+                  "function": {
+                    "type": "string",
+                    "minLength": 1,
+                    "maxLength": 10000,
+                    "description": "Full TypeScript source for optional `type` aliases plus `function scoreScript(expected, actual): number { ... }`. Receives `expected` (the example's expected-output envelope (`{ data: ... }`)) and `actual` (the workflow result envelope (`{ data: ..., files?: ... }`)), returns a number in [0, 1] (the `: number` annotation is required). The `: number` return-type annotation is required and enforced at parse time. Throws are caught and scored as 0."
+                  },
+                  "timeoutMs": {
+                    "default": 5000,
+                    "description": "Maximum wall-clock time the script can run before it is killed and the run is marked failed.",
+                    "type": "integer",
+                    "exclusiveMinimum": 0,
+                    "maximum": 30000
+                  },
+                  "memoryLimitMb": {
+                    "default": 10,
+                    "description": "Maximum memory the sandbox may allocate. Increase if the script processes large arrays or strings.",
+                    "type": "integer",
+                    "exclusiveMinimum": 0,
+                    "maximum": 50
+                  },
+                  "passThreshold": {
+                    "description": "Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model.",
+                    "type": "number",
+                    "minimum": 0,
+                    "maximum": 1
+                  }
+                },
+                "required": [
+                  "function",
+                  "timeoutMs",
+                  "memoryLimitMb"
+                ],
+                "additionalProperties": false,
+                "description": "Configuration for custom-script."
+              }
+            },
+            "required": [
+              "name",
+              "weight",
+              "type",
+              "config"
+            ],
+            "additionalProperties": false
+          }
+        ]
+      }
+    },
+    "passThreshold": {
+      "description": "Workflow-level pass threshold for the weighted mean score. When omitted, legacy evaluator thresholds are combined for compatibility, otherwise the platform default is 0.7.",
+      "type": "number",
+      "minimum": 0,
+      "maximum": 1
+    }
+  },
+  "required": [
+    "evaluators"
+  ],
+  "additionalProperties": false
+}
+```
 <!-- GENERATED:EVALUATOR_CATALOG END -->