@eigenpal/cli 0.16.0 → 0.16.2

package/dist/skill/reference/dataset-format.md

@@ -21,7 +21,7 @@ dataset/
     │   ├── expected.json                   OPTIONAL — expected output or { "$error": ... }
     │   ├── expected/
     │   │   └── Invoice.docx                referenced by expected.json
-    │   └── meta.json                       OPTIONAL — { rowOrder?, annotation?, overrides? }
+    │   └── meta.json                       OPTIONAL — { rowOrder?, annotation?, overrides?, review? }
     │
     └── unsupported-format/
         ├── input.json
@@ -97,6 +97,163 @@ reference is resolved into the S3 file descriptor the worker consumes.
 
 Each file is uploaded to S3 and the original filename is preserved.
 
+## Human review fixtures (`meta.json`)
+
+When a workflow includes `control.human_review`, attach optional `review`
+fixtures to `examples/<name>/meta.json`. Evaluation runs simulate review and
+never pause — fixtures assert selection routing and optional simulated edits.
+
+```json
+{
+  "review": {
+    "version": 1,
+    "steps": {
+      "review-invoice": {
+        "fields": {
+          "/total": {
+            "expectedRoute": "review",
+            "expectedReason": "low_confidence",
+            "expectedValue": 42
+          }
+        },
+        "simulate": {
+          "outcome": "approved",
+          "edits": { "/total": 42 }
+        }
+      }
+    }
+  }
+}
+```
+
+Keys under `steps` match the human review step name. `expectedRoute` is `review`
+or `skip`. `expectedReason` matches selection reason codes such as
+`low_confidence`, `always`, or `threshold_met`. `simulate.outcome` is currently
+`approved`; `simulate.edits` applies reviewer corrections during the simulated
+path. Omit `simulate` to assert routing only.
+
+<!-- GENERATED:DATASET_META_REVIEW START -->
+Schema for `meta.json` → `review` (from `HumanReviewEvaluationFixtureSchema` in `@eigenpal/types`):
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "version": {
+      "type": "number",
+      "const": 1
+    },
+    "steps": {
+      "type": "object",
+      "propertyNames": {
+        "type": "string",
+        "minLength": 1
+      },
+      "additionalProperties": {
+        "type": "object",
+        "properties": {
+          "fields": {
+            "default": {},
+            "type": "object",
+            "propertyNames": {
+              "type": "string"
+            },
+            "additionalProperties": {
+              "type": "object",
+              "properties": {
+                "expectedRoute": {
+                  "type": "string",
+                  "enum": [
+                    "review",
+                    "skip"
+                  ]
+                },
+                "expectedReason": {
+                  "type": "string",
+                  "enum": [
+                    "always",
+                    "explicit",
+                    "low_confidence",
+                    "missing_confidence",
+                    "all",
+                    "reviewer_edit",
+                    "threshold_met",
+                    "never",
+                    "excluded",
+                    "unmatched",
+                    "missing_confidence_skip"
+                  ]
+                },
+                "expectedValue": {
+                  "anyOf": [
+                    {
+                      "type": "string"
+                    },
+                    {
+                      "type": "number"
+                    },
+                    {
+                      "type": "boolean"
+                    },
+                    {
+                      "type": "null"
+                    }
+                  ]
+                }
+              },
+              "additionalProperties": false
+            }
+          },
+          "simulate": {
+            "type": "object",
+            "properties": {
+              "outcome": {
+                "type": "string",
+                "const": "approved"
+              },
+              "edits": {
+                "type": "object",
+                "propertyNames": {
+                  "type": "string"
+                },
+                "additionalProperties": {
+                  "anyOf": [
+                    {
+                      "type": "string"
+                    },
+                    {
+                      "type": "number"
+                    },
+                    {
+                      "type": "boolean"
+                    },
+                    {
+                      "type": "null"
+                    }
+                  ]
+                }
+              }
+            },
+            "additionalProperties": false
+          }
+        },
+        "required": [
+          "fields"
+        ],
+        "additionalProperties": false
+      }
+    }
+  },
+  "required": [
+    "version",
+    "steps"
+  ],
+  "additionalProperties": false
+}
+```
+<!-- GENERATED:DATASET_META_REVIEW END -->
+
 ## Validate before pushing
 
 ```bash
@@ -193,6 +350,7 @@ Failure-expected examples store a single top-level `$error` object. At least one
 | `rowOrder` | integer | no |  | Non-negative display order hint for this example. |
 | `annotation` | string | no |  | Free-form example note, limited to 2000 characters. |
 | `overrides` | record<string, unknown> | no |  | Per-step output overrides as `{ "steps": { "<stepName>": <outputObject> } }`; overridden steps are skipped or partially merged during evaluation. |
+| `review` | object | no |  | Optional human-review evaluation fixture: expected field routes and simulated approvals or edits for this example. |
 
 
 ### Import mode (`DatasetImportModeSchema`)
@@ -238,6 +396,121 @@ Allowed values: `"append"` \| `"replace"`
         "type": "string"
       },
       "additionalProperties": {}
+    },
+    "review": {
+      "description": "Optional human-review evaluation fixture: expected field routes and simulated approvals or edits for this example.",
+      "type": "object",
+      "properties": {
+        "version": {
+          "type": "number",
+          "const": 1
+        },
+        "steps": {
+          "type": "object",
+          "propertyNames": {
+            "type": "string",
+            "minLength": 1
+          },
+          "additionalProperties": {
+            "type": "object",
+            "properties": {
+              "fields": {
+                "default": {},
+                "type": "object",
+                "propertyNames": {
+                  "type": "string"
+                },
+                "additionalProperties": {
+                  "type": "object",
+                  "properties": {
+                    "expectedRoute": {
+                      "type": "string",
+                      "enum": [
+                        "review",
+                        "skip"
+                      ]
+                    },
+                    "expectedReason": {
+                      "type": "string",
+                      "enum": [
+                        "always",
+                        "explicit",
+                        "low_confidence",
+                        "missing_confidence",
+                        "all",
+                        "reviewer_edit",
+                        "threshold_met",
+                        "never",
+                        "excluded",
+                        "unmatched",
+                        "missing_confidence_skip"
+                      ]
+                    },
+                    "expectedValue": {
+                      "anyOf": [
+                        {
+                          "type": "string"
+                        },
+                        {
+                          "type": "number"
+                        },
+                        {
+                          "type": "boolean"
+                        },
+                        {
+                          "type": "null"
+                        }
+                      ]
+                    }
+                  },
+                  "additionalProperties": false
+                }
+              },
+              "simulate": {
+                "type": "object",
+                "properties": {
+                  "outcome": {
+                    "type": "string",
+                    "const": "approved"
+                  },
+                  "edits": {
+                    "type": "object",
+                    "propertyNames": {
+                      "type": "string"
+                    },
+                    "additionalProperties": {
+                      "anyOf": [
+                        {
+                          "type": "string"
+                        },
+                        {
+                          "type": "number"
+                        },
+                        {
+                          "type": "boolean"
+                        },
+                        {
+                          "type": "null"
+                        }
+                      ]
+                    }
+                  }
+                },
+                "additionalProperties": false
+              }
+            },
+            "required": [
+              "fields"
+            ],
+            "additionalProperties": false
+          }
+        }
+      },
+      "required": [
+        "version",
+        "steps"
+      ],
+      "additionalProperties": false
     }
   },
   "additionalProperties": false

package/dist/skill/reference/evaluators.md

@@ -356,7 +356,7 @@ _Generated from `EvalConfigYamlSchema` in `@eigenpal/types/src/eval/evaluator-co
 | Field | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
 | `model` | string | no |  | Which LLM grades the output. Falls back to the workspace's default LLM provider when unset. |
-| `reasoningEffort` | `"none"` \| `"minimal"` \| `"low"` \| `"medium"` \| `"high"` \| `"xhigh"` \| `"max"` | no |  | Reasoning effort for models that support it. Omit to preserve the current provider default. |
+| `reasoningEffort` | `"none"` \| `"minimal"` \| `"low"` \| `"medium"` \| `"high"` \| `"xhigh"` \| `"max"` | no |  | Reasoning effort for models that support it. Omit to use the selected model's default. |
 | `mode` | `"continuous"` \| `"discrete"` | no | `"continuous"` | Continuous = the LLM returns a free-form score in [0, 1]. Discrete = the LLM picks one of your labels and the score is looked up from the table. |
 | `labels` | record<string, number> | no |  | Allowed labels and the score each maps to. Required in discrete mode. The judge MUST return one of these labels; each score must be in [0, 1]. |
 | `passThreshold` | number | no |  | Legacy per-evaluator pass threshold. Pass/fail now uses the single workflow-level pass threshold; this is read only to preserve the run gate of configs authored before the single-threshold model. |
@@ -562,7 +562,7 @@ Generated from the same discriminated union used by evaluator validation; YAML k
                     "minLength": 1
                   },
                   "reasoningEffort": {
-                    "description": "Reasoning effort for models that support it. Omit to preserve the current provider default.",
+                    "description": "Reasoning effort for models that support it. Omit to use the selected model's default.",
                     "type": "string",
                     "enum": [
                       "none",