@eigenpal/cli 0.20.43 → 0.20.45

package/dist/skill/SKILL.md

@@ -132,6 +132,39 @@ Use `output.textFormat` for representation and `output.require` for required evi
 
 Legacy `ai.parse` remains supported. Never mix its flat `parseMode`/`nativeText` settings into v2. Migrate to a new file with `eigenpal workflow migrate-parser workflow.yaml --policy preserve --out workflow-v2.yaml` or `--policy auto` to adopt recommended defaults. Preserve mode retains OCR-only restrictions; automatic mode can broaden egress and costs. Review reported changes and representative outputs before publishing.
 
+## Dataset ground truth and targeted reviews
+
+Dataset `expected.json` must match the complete intended workflow output
+one-to-one: the same fields, nesting, arrays, value types, and expected file
+references. Keep the correct expected values, even when the workflow currently
+returns something wrong. Never replace ground truth with the current run output
+just to make an evaluator pass, and never add evaluator-only summaries, flattened
+projections, comparison metadata, or reviewer instructions to expected output.
+Evaluator configuration owns which outputs are cross-checked and how, including
+ignored fields, tolerances, array matching, and custom comparisons.
+
+A dataset review snapshots that complete expected output unchanged. Narrow the
+review with flags and comments, never by pruning or reshaping `expected.json`:
+
+- Use `--focus` or `--focus-reason` only for specific uncertain or incorrect
+  leaf paths, for example `items.3.amount`. A parent such as `items` flags every
+  descendant, potentially thousands of fields. Do not enumerate all fields as
+  focus paths unless the user explicitly requests a full-field review.
+- Focus paths apply to every selected example. Group examples with the same
+  concerns into a request; create separate requests when their focus differs.
+- Use `--field-note 'invoice-foo.items.3.amount=Workflow currently returns 12.50; expected 10.50. Verify the source line.'`
+  to attach evidence to that exact field. Notes do not flag fields: add a focus
+  path separately when it needs a decision. Preserve the actual value as a
+  comment, not as replacement ground truth.
+- For an existing review, use `item --action comment --field-path items.3.amount
+  --comment 'Workflow currently returns 12.50; expected 10.50. Verify the source line.'
+  --expected-updated-at <iso>`. Use `update --focus-json '[{"path":"items.3.amount","reason":"Amount differs from source"}]'`
+  to replace an overly broad focus list, or `--focus-json '[]'` to clear it.
+- Before sharing a review, inspect `get --json`: verify the complete expected
+  output, a short justified `focusFields` list, and field-specific note events.
+  An unflagged field remains visible for context; it does not need a flag merely
+  because it appears in the dataset. Never approve on the human reviewer's behalf.
+
 ## Workflow Loop
 
 ```bash
@@ -188,7 +221,7 @@ eigenpal workflow evaluators push <workflow-id> --file ./evaluators.yaml
 eigenpal workflow dataset review-request create <workflow-id> \
   --title "<topic> GT review" --example-name <example-name> \
   --focus <path> --focus-reason '<path>=why' --ignore <path> \
-  --item-note '<example>=note' --field-note '<example.path>=note' \
+  --item-note '<example>=note' --field-note '<example.path>=Workflow currently returns <actual>; expected <expected>. Verify against the source.' \
   --status open --json
 # Poll .progress.complete until reviewers finish. Record field decisions and
 # approve/reject with comments as needed, then pull and MANUAL reconcile —

package/dist/skill/reference/cli/agents.md

@@ -725,21 +725,21 @@ Request human review of dataset ground truth. Snapshot examples and poll progres
 
 ### Options
 
-| Flag                             | Required | Default | Description                                                                      |
-| -------------------------------- | -------- | ------- | -------------------------------------------------------------------------------- |
-| `--base-url <url>`               | no       |         | Server base URL                                                                  |
-| `--json`                         | no       |         | Emit machine-readable JSON on stdout                                             |
-| `--title <title>`                | yes      |         | Review request title                                                             |
-| `--example-name <name>`          | yes      | `[]`    | Example folder name to include (repeatable)                                      |
-| `--instructions <text>`          | no       |         | Note shown to the reviewer for the whole request                                 |
-| `--focus <path>`                 | no       | `[]`    | Expected-output path to highlight (repeatable)                                   |
-| `--focus-reason <spec>`          | no       | `[]`    | Reason a focus path needs review, as path=reason (repeatable)                    |
-| `--ignore <path>`                | no       | `[]`    | Expected-output path reviewers can skip (repeatable)                             |
-| `--item-note <spec>`             | no       | `[]`    | Example-level note seeded at create time, as exampleName=comment (repeatable)    |
-| `--field-note <spec>`            | no       | `[]`    | Field-level note seeded at create time, as exampleName.path=comment (repeatable) |
-| `--focus-json <json>`            | no       |         | JSON array of { path, reason? } focus fields                                     |
-| `--notes-json <json>`            | no       |         | JSON array of { exampleName, comment?, fields?: [{ path, comment }] }            |
-| `--status <draft\|open\|paused>` | no       |         | Initial lifecycle status (default: draft)                                        |
+| Flag                             | Required | Default | Description                                                                                                 |
+| -------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------- |
+| `--base-url <url>`               | no       |         | Server base URL                                                                                             |
+| `--json`                         | no       |         | Emit machine-readable JSON on stdout                                                                        |
+| `--title <title>`                | yes      |         | Review request title                                                                                        |
+| `--example-name <name>`          | yes      | `[]`    | Example folder name to include (repeatable)                                                                 |
+| `--instructions <text>`          | no       |         | Note shown to the reviewer for the whole request                                                            |
+| `--focus <path>`                 | no       | `[]`    | Expected-output leaf path to flag across selected examples; parent paths flag every descendant (repeatable) |
+| `--focus-reason <spec>`          | no       | `[]`    | Reason a focus path needs review, as path=reason (repeatable)                                               |
+| `--ignore <path>`                | no       | `[]`    | Expected-output path reviewers can skip (repeatable)                                                        |
+| `--item-note <spec>`             | no       | `[]`    | Example-level note seeded at create time, as exampleName=comment (repeatable)                               |
+| `--field-note <spec>`            | no       | `[]`    | Field comment only, as exampleName.path=comment; does not flag the field (repeatable)                       |
+| `--focus-json <json>`            | no       |         | JSON array of { path, reason? } focus fields                                                                |
+| `--notes-json <json>`            | no       |         | JSON array of { exampleName, comment?, fields?: [{ path, comment }] }                                       |
+| `--status <draft\|open\|paused>` | no       |         | Initial lifecycle status (default: draft)                                                                   |
 
 ### `eigenpal agents dataset review-request get [options] <automation-id> <review-id>`
 

package/dist/skill/reference/cli/workflow.md

@@ -701,21 +701,21 @@ Request human review of dataset ground truth. Snapshot examples and poll progres
 
 ### Options
 
-| Flag                             | Required | Default | Description                                                                      |
-| -------------------------------- | -------- | ------- | -------------------------------------------------------------------------------- |
-| `--base-url <url>`               | no       |         | Server base URL                                                                  |
-| `--json`                         | no       |         | Emit machine-readable JSON on stdout                                             |
-| `--title <title>`                | yes      |         | Review request title                                                             |
-| `--example-name <name>`          | yes      | `[]`    | Example folder name to include (repeatable)                                      |
-| `--instructions <text>`          | no       |         | Note shown to the reviewer for the whole request                                 |
-| `--focus <path>`                 | no       | `[]`    | Expected-output path to highlight (repeatable)                                   |
-| `--focus-reason <spec>`          | no       | `[]`    | Reason a focus path needs review, as path=reason (repeatable)                    |
-| `--ignore <path>`                | no       | `[]`    | Expected-output path reviewers can skip (repeatable)                             |
-| `--item-note <spec>`             | no       | `[]`    | Example-level note seeded at create time, as exampleName=comment (repeatable)    |
-| `--field-note <spec>`            | no       | `[]`    | Field-level note seeded at create time, as exampleName.path=comment (repeatable) |
-| `--focus-json <json>`            | no       |         | JSON array of { path, reason? } focus fields                                     |
-| `--notes-json <json>`            | no       |         | JSON array of { exampleName, comment?, fields?: [{ path, comment }] }            |
-| `--status <draft\|open\|paused>` | no       |         | Initial lifecycle status (default: draft)                                        |
+| Flag                             | Required | Default | Description                                                                                                 |
+| -------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------- |
+| `--base-url <url>`               | no       |         | Server base URL                                                                                             |
+| `--json`                         | no       |         | Emit machine-readable JSON on stdout                                                                        |
+| `--title <title>`                | yes      |         | Review request title                                                                                        |
+| `--example-name <name>`          | yes      | `[]`    | Example folder name to include (repeatable)                                                                 |
+| `--instructions <text>`          | no       |         | Note shown to the reviewer for the whole request                                                            |
+| `--focus <path>`                 | no       | `[]`    | Expected-output leaf path to flag across selected examples; parent paths flag every descendant (repeatable) |
+| `--focus-reason <spec>`          | no       | `[]`    | Reason a focus path needs review, as path=reason (repeatable)                                               |
+| `--ignore <path>`                | no       | `[]`    | Expected-output path reviewers can skip (repeatable)                                                        |
+| `--item-note <spec>`             | no       | `[]`    | Example-level note seeded at create time, as exampleName=comment (repeatable)                               |
+| `--field-note <spec>`            | no       | `[]`    | Field comment only, as exampleName.path=comment; does not flag the field (repeatable)                       |
+| `--focus-json <json>`            | no       |         | JSON array of { path, reason? } focus fields                                                                |
+| `--notes-json <json>`            | no       |         | JSON array of { exampleName, comment?, fields?: [{ path, comment }] }                                       |
+| `--status <draft\|open\|paused>` | no       |         | Initial lifecycle status (default: draft)                                                                   |
 
 ### `eigenpal workflow dataset review-request get [options] <automation-id> <review-id>`
 

package/dist/skill/reference/dataset-format.md

@@ -42,8 +42,16 @@ are explicit references:
 }
 ```
 
-`expected.json` mirrors the expected automation output. Expected files use the
-same reference shape with the `expected/` prefix.
+`expected.json` must mirror the complete intended automation output one-to-one:
+fields, nesting, array structure, value types, and expected file references.
+Expected files use the same reference shape with the `expected/` prefix.
+Use the correct expected values even when the workflow currently returns wrong
+values. Evaluators define which parts to cross-check and how (tolerances, ignored
+fields, array matching, custom comparison); do not flatten, summarize, omit fields,
+or add comparison-only metadata to ground truth to suit evaluator logic.
+Dataset reviews snapshot that full expected output unchanged. Review scope and
+reviewer evidence belong in focus paths and field comments, not in a reduced or
+augmented expected-output object.
 
 ## Rules
 
@@ -327,6 +335,23 @@ There is **no auto-apply** by design: after review, `dataset pull` and
 manually reconcile each example into the live dataset (reviewers can err).
 Close via `update --status closed`.
 
+Flag only uncertain or incorrect leaf paths with `--focus` / `--focus-reason`.
+A parent path such as `items` flags every descendant; prefer `items.3.amount`
+when only that value needs checking. Focus is shared across the selected examples;
+use separate requests for examples with different concerns. Do not flag every
+field just because the complete dataset is visible. An empty focus list flags
+nothing, while all output fields remain available as context.
+
+`--field-note 'invoice-foo.items.3.amount=Workflow currently returns 12.50; expected 10.50. Verify the source line.'`
+leaves a comment on that exact example and field without flagging it. Add a focus
+path separately if it needs review. For an existing request, use `item --action
+comment --field-path items.3.amount --comment 'Workflow currently returns 12.50; expected 10.50.'
+--expected-updated-at <iso>`. Read the item first to obtain its timestamp.
+Inspect `get --json` before sharing: check the full expected output, the intended
+focus list, and field note events. To repair excessive flags, replace the list
+with `update --focus-json '[{"path":"items.3.amount","reason":"Amount differs from source"}]'`;
+`--focus-json '[]'` clears it. This changes review scope, not dataset content.
+
 ```bash
 # Ask a reviewer to inspect specific fields, with reasons, then poll until done.
 eigenpal workflow dataset review-request create <automation-id> \
@@ -335,7 +360,7 @@ eigenpal workflow dataset review-request create <automation-id> \
   --instructions "Check IBAN checksums and invoice totals." \
   --focus vendor.iban --focus-reason 'vendor.iban=OCR often mangles IBANs' \
   --ignore currency \
-  --field-note 'invoice-foo.total=off by 0.01 last run' \
+  --field-note 'invoice-foo.vendor.iban=Workflow currently returns SK12; expected SK34. Verify against the document.' \
   --status open --json
 
 # List requests; `.progress.complete` means every example was decided.

package/dist/skill/reference/step-types.md

@@ -8294,7 +8294,7 @@ Pause the run for selective field confirmation or correction, then continue with
 | --- | --- | --- | --- | --- |
 | `data` | string \| record<string, unknown> \| array<unknown> | yes |  | Structured data or a template expression resolving to an object or array |
 | `schema` | record<string, unknown> | no |  | Optional JSON Schema for validating edits and exposing downstream autocomplete. When omitted with metadataFrom: ai.extract and a direct whole-output data expression, the extract step schema is reused. |
-| `metadataFrom` | `"ai.extract"` | no |  | Producer adapter. `ai.extract` derives field confidence from `_grounding` and strips that reserved key from review data. Grounding is not stripped unless this is set. |
+| `metadataFrom` | `"ai.extract"` | no |  | Producer adapter. `ai.extract` derives categorical field confidence from `_grounding`, copies valid numeric grounding scores into fieldMetadata.display.score, and strips that reserved key from review data. Grounding is not stripped unless this is set. |
 | `fieldMetadata` | record<string, object> | no | `{}` | RFC 6901 pointer map of producer facts (labels, confidence, display). Selection policy belongs under selection.fields; fieldMetadata.review is a legacy alias. |
 | `selection` | object | yes |  | Field selection policy resolved once when the task is created |
 | `files` | object | no | `{"includeRunInputs":true,"attachments":[]}` | Run input files and optional current-run attachments for reviewers |
@@ -8346,7 +8346,7 @@ Config schema:
       "enum": [
         "ai.extract"
       ],
-      "description": "Producer adapter. `ai.extract` derives field confidence from `_grounding` and strips that reserved key from review data. Grounding is not stripped unless this is set."
+      "description": "Producer adapter. `ai.extract` derives categorical field confidence from `_grounding`, copies valid numeric grounding scores into fieldMetadata.display.score, and strips that reserved key from review data. Grounding is not stripped unless this is set."
     },
     "fieldMetadata": {
       "description": "RFC 6901 pointer map of producer facts (labels, confidence, display). Selection policy belongs under selection.fields; fieldMetadata.review is a legacy alias.",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eigenpal/cli",
-  "version": "0.20.43",
+  "version": "0.20.45",
   "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
   "keywords": [
     "eigenpal",