@uipath/skills 1.201.0-preview.457 → 1.201.0-preview.488

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@uipath/skills",
-  "version": "1.201.0-preview.457",
+  "version": "1.201.0-preview.488",
   "description": "UiPath agent skills for Claude Code, Codex, Cursor, Copilot, Gemini and OpenCode — RPA, UI automation, UI testing, coded agents/apps/workflows, and troubleshooting. Distributed as the UiPath Claude Code plugin.",
   "author": {
     "name": "UiPath"

package/skills/uipath-ixp/SKILL.md

@@ -99,7 +99,7 @@ If the user provides a taxonomy file, use `--skip-taxonomy` and `import-taxonomy
 | "Confirm one line item / extraction" / "Confirm only this occurrence" | `uip ixp labellings confirm <project-name> <document-id> --group <name> --occurrence <N> [--fields <ids>] --output json` — targets one specific extraction of a repeatable field group (0-based index **from the latest `get-predictions`**). Without `--fields`, confirms every predicted field in that occurrence; with `--fields`, only those. Other occurrences untouched. Confirming renumbers the group on the next read (the confirmed row moves to `Occurrence` 0) — so batch multiple occurrences into one `--updates '[…]'` call rather than chaining `--occurrence` calls off a single read. See Critical Rules 13 and 18. |
 | "Unconfirm one line item / extraction" / "Roll back only this occurrence" | `uip ixp labellings unconfirm <project-name> <document-id> --group <name> --occurrence <N> [--fields <ids>] --output json` — rolls back one specific extraction of a repeatable field group (0-based index, same as `get-predictions`/`confirm`). Without `--fields`, unconfirms every annotated field in that occurrence; with `--fields`, only those. Other occurrences untouched. **Re-read `get-predictions` first** — on a partly-confirmed group the confirmed rows sort to the front, so the index that confirmed a row is usually not the index that rolls it back (Critical Rule 18). For several occurrences in one call, use `--updates '[…]'` instead. See Critical Rule 14. |
 | "Set overall extraction instructions" / "Update project prompt" | `uip ixp projects update-prompt <project-name> --prompt "<text>" --output json` — replaces the taxonomy-wide prompt (the "Overall extraction instructions" field in the IXP UI). Distinct from `fields update-prompts` (per-field) and `groups update-prompts` (per-field-group). |
-| "How is this project performing?" / "What's the F1?" | Resolve the live version with `uip ixp projects list-models <project-name> --output json`, then `uip ixp projects get-metrics <project-name> --model-version <live-version> --output json` (Critical Rule 21). If Data is `{ Metrics: null }` the model isn't validated yet — report that and stop. If the call instead returns `Result: Failure` with `ErrorCode: not_found` (a project with no trained model yet, e.g. no confirmed labellings), treat it the same way — report "no metrics yet" and stop. Otherwise Data is flat; report in order: (1) which version the scores belong to + `TrainedTime`; (2) overall `ProjectScore`/`ProjectScoreQuality`; (3) per-group scores from `FieldGroups[]` (F1/Precision/Recall); (4) per-field scores from `Fields[]`, sorted lowest-F1 first. State numbers plainly; no "good enough" judgement unless asked; route low scores to [Improve Prompts Guide](references/improve-prompts-guide.md). Answer from these calls only — no ad-hoc discovery (Critical Rule #1). |
+| "How is this project performing?" / "What's the F1?" | Resolve the live version with `uip ixp projects list-models <project-name> --output json`, then `uip ixp projects get-metrics <project-name> --model-version <live-version> --output json` (Critical Rule 21). If Data is `{ Metrics: null }` the model isn't validated yet — report that and stop. If the call instead returns `Result: Failure` with `ErrorCode: not_found` (a project with no trained model yet, e.g. no confirmed labellings), treat it the same way — report "no metrics yet" and stop. Otherwise Data is flat; report in order: (1) which version the scores belong to + `TrainedTime`; (2) overall `ProjectScore`/`ProjectScoreQuality`; (3) per-group scores from `FieldGroups[]` (F1/Precision/Recall); (4) per-field scores from `Fields[]`, sorted lowest-F1 first — F1 **with** its `Precision`/`Recall` (a low F1 means the opposite fix depending on which side is short), plus `Annotations` (the sample size behind the F1) and `ErrorRate` (= errors/`Annotations` — it counts misses, so it is **not** `1 - Precision`). Ignore the `Quality` labels (derived, inconsistent scales — [Improve Prompts Guide § What get-metrics returns](references/improve-prompts-guide.md#what-get-metrics-returns-and-which-values-decide)). State numbers plainly; no "good enough" judgement unless asked; route low scores to [Improve Prompts Guide](references/improve-prompts-guide.md). Answer from these calls only — no ad-hoc discovery (Critical Rule #1). |
 | "Describe this project" / "What's in it?" | Three calls, reported in order: (1) identity — `Title`/`Name` from `uip ixp projects get <project-name> --output json`; (2) current model **version** — live/published + `TrainedTime` from `list-models` (the trained version, not the configured extraction model — for that see the row above); (3) taxonomy — label-group/field counts from `uip ixp projects get-taxonomy <project-name> --output json` (raw artifact: counts live under `Data.dataset.label_groups` and `Data.dataset.entity_defs`, snake_case). Fold in performance (above) only if asked. Do NOT page `documents list` (its `Data` is a paged `{ Documents, Total, Offset, Limit }` envelope — use `Total` for a count) or read deployment bindings. Answer from these calls only (Critical Rule #1). |
 
 ## Common Pitfalls

package/skills/uipath-ixp/references/cli-reference.md

@@ -15,7 +15,7 @@ All commands use `uip ixp` prefix. Always append `--output json` when parsing ou
 | `uip ixp projects update-title <project-name> "<new-title>" --output json` | Update the display title of a project |
 | `uip ixp projects update-prompt <project-name> --prompt "<text>" --output json` | Update the project's **Overall extraction instructions** — the taxonomy-wide prompt the model sees on every extraction (the field at the top of the IXP UI's Manage Taxonomy page). Distinct from per-field-group prompts (`groups update-prompts`) and per-field prompts (`fields update-prompts`). Replaces the existing value. |
 | `uip ixp projects get-taxonomy <project-name> --output json` | Export the raw IXP taxonomy artifact. Data is `{ status, dataset: { entity_defs, label_groups } }` — read `entity_defs` and `label_groups` under `dataset`. Intended for re-import (see `import-taxonomy`), not a human-readable view. `dataset` also carries `_model_config`, the only read path for the configured extraction model and pre-processing — see [Reading the current model and pre-processing](#reading-the-current-model-and-pre-processing). |
-| `uip ixp projects get-metrics <project-name> [--model-version <N>] --output json` | Get validation metrics. **Validated model →** flat Data: `ProjectScore`, `ProjectScoreQuality`, `ValidatedDocuments`, `ModelVersion`, plus per-group `FieldGroups[]` (`FieldGroup`, `F1`, `Precision`, `Recall`, `ErrorRate`, `Documents`) and per-field `Fields[]` (`FieldGroup`, `FieldId`, `F1`, `Precision`, `Recall`, `ErrorRate`, `Documents`, `Annotations`, `Quality`). **Trained but not yet validated →** Data is `{ Metrics: null }` (not an error). **No trained model yet (e.g. a project with no confirmed labellings) →** the call returns a failure envelope `Result: Failure` with `ErrorCode: not_found` (no `Data`), NOT `{ Metrics: null }` — treat it as "no metrics yet". **Defaults to the LATEST TRAINED version, which is NOT necessarily the published/live one** — resolve the version from `list-models` and pass it as `--model-version <N>` whenever you report a score, so the numbers and the version identity match (SKILL.md Critical Rule 21). |
+| `uip ixp projects get-metrics <project-name> [--model-version <N>] --output json` | Get validation metrics. **Validated model →** flat Data: `ProjectScore`, `ProjectScoreQuality`, `ValidatedDocuments`, `ModelVersion`, plus per-group `FieldGroups[]` (`FieldGroup`, `F1`, `Precision`, `Recall`, `ErrorRate`, `Documents`) and per-field `Fields[]` (`FieldGroup`, `FieldId`, `F1`, `Precision`, `Recall`, `ErrorRate`, `Documents`, `Annotations`, `Quality`). **Trained but not yet validated →** Data is `{ Metrics: null }` (not an error). **No trained model yet (e.g. a project with no confirmed labellings) →** the call returns a failure envelope `Result: Failure` with `ErrorCode: not_found` (no `Data`), NOT `{ Metrics: null }` — treat it as "no metrics yet". **Defaults to the LATEST TRAINED version, which is NOT necessarily the published/live one** — resolve the version from `list-models` and pass it as `--model-version <N>` whenever you report a score, so the numbers and the version identity match (SKILL.md Critical Rule 21). Field semantics — which values decide and which are derived — are in [Improve Prompts Guide § What get-metrics returns](improve-prompts-guide.md#what-get-metrics-returns-and-which-values-decide). `ErrorRate` is `errors / Annotations` (it counts misses — not `1 - Precision`); the `Quality`/`ProjectScoreQuality` labels use inconsistent scales — never gate on them. |
 | `uip ixp projects configure-model <project-name> [options] --output json` | Configure extraction model. Options: `--model` (gemini_2_5_flash/gemini_2_5_pro/gpt_4o_2024_05_13) and `--preprocessing` (none/table_mini/table). To read the current settings, see [Reading the current model and pre-processing](#reading-the-current-model-and-pre-processing). |
 | `uip ixp projects list-models <project-name> --output json` | List all model versions and tags. Returns `Models[]` (`Version`, `ModelName`, `Pinned`, `TrainedTime`, `Description`), `Tags[]` (`Name`, `Version`, `UpdatedAt`), and `MaxPublished`. **The only read path for the project's live version** — `Tags[]` entry Name=`live`, else the highest `Models[]` with `Pinned: true`; which version a **folder** serves at runtime is a different question — [Deployments](#deployments). `ModelName` is the trained labeller's **family** (e.g. `gemini_ixp`, `gemini_pro_ixp`) — it is never a `--model` value like `gemini_2_5_flash`, so it does not answer "which extraction model is configured" (see [Reading the current model and pre-processing](#reading-the-current-model-and-pre-processing)). |
 | `uip ixp projects publish <project-name> [--model-version <N>] [--tag <live\|staging>] --output json` | Publish a model version — defaults to the latest; pass `-m, --model-version <N>` to pick a specific one. `-d, --description "<text>"` sets a description; `--tag <live\|staging>` tags the published version. |

package/skills/uipath-ixp/references/improve-prompts-guide.md

@@ -32,12 +32,32 @@ Each command sends one server-side call; the server matches by name and writes p
 
 The user may specify a max number of iterations (default: 3). Track:
 
-- **Baseline metrics** — the per-field F1 scores before any changes
-- **Previous iteration metrics** — the per-field F1 scores from the last successful iteration
+- **Baseline metrics** — the `get-metrics` payload before any changes, and its `ModelVersion` — a trained version's metrics can be re-read at any time with `--model-version <N>`, so keeping the version number is enough to recover anything. The values that drive the loop are mapped in [What `get-metrics` returns](#what-get-metrics-returns-and-which-values-decide); the rest is reported once or ignored.
+- **Previous iteration metrics** — the same, for the last successful iteration's version
 - **Previous instructions** — the per-field (field) instructions from the last successful iteration (for rollback)
 
 Do NOT re-read the taxonomy or sample documents between iterations — use what you already have. Only re-read metrics after each instruction update + retrain cycle. This assumes no one modifies the taxonomy or documents externally during the loop. If the user mentions changes were made in the web UI, re-fetch the taxonomy and document list before continuing.
 
+## What `get-metrics` returns, and which values decide
+
+The values `get-metrics` returns are neither independent nor interchangeable — using the wrong one silently changes the loop's behaviour.
+
+| Value | Level | Role in this loop |
+|-------|-------|-------------------|
+| `F1` | field, group | **Decision variable.** Targeting (2a), regression (2f), stopping (2f). |
+| `Precision` | field, group | **Decision variable.** Splits a low `F1` into PRECISION vs RECALL (2a) — that split picks which rewrite to attempt. |
+| `Recall` | field, group | **Decision variable.** Same split, plus the `< 0.5` labelling-gap probe (2a-check). |
+| `Annotations` | field | **Decision variable.** Reviewed **extractions** for that field (not documents) — the sample `F1` is computed over, so it sets that field's regression threshold (2f). |
+| `Documents` | field, group | **Decision variable.** How many documents this field (or group) was reviewed in — a per-field count, not a project total. `0` → SKIP (2a): no evidence to evaluate a rewrite against. Below `ValidatedDocuments` → some reviewed documents carry no label for this field (2f). |
+| `ProjectScore` | project | **Report only** — the headline number, an average of the per-field `F1` values. Never gate on it (2f diffs the fields directly). |
+| `ValidatedDocuments` | project | **Decision variable.** How many labelled documents the metrics are computed over — project-level only, and the ceiling for every per-field `Documents`. Below the project's total document count → unlabelled documents exist; label them before looping (1e). |
+| `ModelVersion` | project | **Decision variable.** Retrain completion ([Waiting for retrain](#waiting-for-retrain)). |
+| `ErrorRate` | field, group | **Report — independent of `Precision`.** Wrong extractions over `Annotations`. A wrong value counts **once** (not as a false positive plus a false miss), and a miss counts even though it cannot lower `Precision` — so `Precision` 1.00 can still carry `ErrorRate` 0.20. Report it as the manual-correction burden; diagnose direction from `Precision`/`Recall`. |
+| `Quality` | field | **Ignore.** A coarse label derived from the numbers, on a scale inconsistent with `ProjectScoreQuality` (an `F1` of 1.00 still reads `good` while a `ProjectScore` of 0.91 reads `excellent`). Never gate on it and don't report it per field — if the user asks about the UI's label, explain the scales differ. |
+| `ProjectScoreQuality` | project | **Report on the project line only** (the label the UI shows beside the score) — different scale from field `Quality` (above). |
+| `FieldGroup`, `FieldId` | field | Identity. `FieldId` needs the taxonomy join for a human-readable name (see 1a). |
+
+
 ## Waiting for retrain
 
 Every change to model inputs — labellings, instructions, document upload/delete, taxonomy edits — triggers a full retrain. Metrics read mid-retrain are *pre*-change scores and corrupt every downstream comparison, so wait before each metrics read.
@@ -131,12 +151,14 @@ Use the current metrics (baseline on first iteration, post-relabel metrics on su
    - `Documents < 1` → **SKIP**
    - Otherwise → **REFINE**
 
-2. **Diagnose the problem type** (use F1 as primary signal with few documents):
+2. **Diagnose the problem type** from the `Precision`/`Recall` split — `F1` says *how bad*, the split says *what to write*:
    - `Precision < Recall` significantly → **PRECISION** — model extracts wrong values
    - `Recall < Precision` significantly → **RECALL** — model misses the field
    - Otherwise → **BOTH** — rewrite entirely
 
-Print a diagnosis summary showing each field group's score and each field's name, F1, precision/recall, and diagnosis.
+3. **Record the field's `Annotations` count** next to the diagnosis. It does not change the classification, but it sets how much of the following delta you are entitled to believe (2f), so carry it forward rather than re-fetching it later.
+
+Print a diagnosis summary with one row per field — name, `F1`, `Precision`, `Recall`, `ErrorRate`, `Annotations`, `Documents`, diagnosis — plus the group rows and the project line (`ProjectScore` / `ProjectScoreQuality` / `ValidatedDocuments` / `ModelVersion`). Ignore the `Quality` labels ([What `get-metrics` returns](#what-get-metrics-returns-and-which-values-decide)).
 
 If no fields need REFINE, stop — the project is already at target quality.
 
@@ -240,13 +262,41 @@ If `ModelVersion` hasn't advanced since the last check, keep re-reading under th
 
 ### 2f. Compare and decide
 
-Compare the new per-field F1 scores against the **previous iteration** scores.
+Compare the new metrics against the **previous iteration** at both levels — the fields you touched, and the project as a whole.
+
+#### Regression noise floor
+
+With few `Annotations`, `F1` moves in jumps: a single annotation flipping by chance jumps it as far as a genuinely worse instruction would, and the number alone cannot tell the two apart. The rollback threshold therefore scales with the sample:
+
+```text
+regression_threshold = max(0.1, 1 / Annotations)
+```
+
+That is 0.2 at `Annotations` = 5 — one flipped annotation is not evidence — and the flat 0.1 from `Annotations` = 10 up. Fields whose `Annotations` differ get different thresholds in the same iteration; that is intended, not an inconsistency.
+
+**Below the threshold is not "no change" — it is "not measurable yet".** Do not report a sub-threshold move as an improvement either. If a field keeps drifting sub-threshold across iterations and its `Annotations` is small, no prompt rewrite can be evaluated — but which remedy to report depends on *why* the sample is small.
+
+**A small `Annotations` has two causes with opposite remedies.** `Annotations` counts reviewed **extractions**, not documents — one document can contribute several — so it cannot be compared against a document count directly. Compare the field's own `Documents` against the project-level `ValidatedDocuments`:
+
+- **`Documents` equal to `ValidatedDocuments`** → this field already has evidence on every labelled document; the sample is as large as the data allows. Tag it **UPLOAD**.
+- **`Documents` below `ValidatedDocuments`** → some labelled documents carry no evidence for this field, and the payload cannot say why — never reviewed there, or reviewed and skipped because the prediction was wrong. Tag it **REVIEW** — the review pass ([Label Documents Guide](label-documents-guide.md)) shows which in seconds, and 2a-check's `Recall < 0.5` gate would never trigger it. Even when the review finds nothing to add, confirming that costs a glance, while an unreviewed document left unfound caps the field for good.
+
+Both tags are **final-report lines, not loop actions**: the loop runs on to its normal stopping criteria — never pause mid-run to ask for documents or to review — and the report then says plainly that a tagged field's score cannot rise further until its sample grows.
 
-**Selective regression check:** For each field you updated this iteration, check if F1 dropped by more than 0.1:
+`Annotations / Documents` is the average number of extractions per document — about 1 for a single-value field, higher under a repeatable group.
 
-- **Regressed fields** (F1 dropped >0.1): roll back ONLY those fields' instructions to the previous iteration's version. Keep the improved instructions for fields that gained or held steady.
+**Selective regression check:** For each field you updated this iteration, compare its `F1` drop against **that field's** `regression_threshold`:
+
+- **Regressed fields** (drop > their threshold): roll back ONLY those fields' instructions to the previous iteration's version. Keep the improved instructions for fields that gained or held steady.
 - **Improved/unchanged fields**: keep their new instructions.
 
+**Collateral check (fields you did NOT touch):** per-field checks only cover the fields you edited, but a `groups update-prompts` edit rewrites the parent `label_def` and so moves every field under it.
+
+Do **not** gate this on `ProjectScore`. It is an average over fields — observed to be the unweighted mean of the per-field `F1` values — so it carries nothing the `Fields[]` array does not, and it divides a single field's move by the field count, burying a real regression below its own noise. Diff **every** field against the previous iteration instead, each against **its own** `regression_threshold`:
+
+- An **edited** field regressed beyond its threshold → roll that field back, as above.
+- An **unedited** field regressed beyond its threshold → collateral damage. Report it by name with its delta. Roll it back only when it shares a field group with a `groups update-prompts` edit you made this iteration — that is the one interaction with a mechanical cause. Otherwise **keep the iteration and re-check next round**: two metric reads cannot establish that your edit caused the move, and discarding edits that individually passed destroys work on a guess.
+
 If any fields regressed, do a selective rollback:
 
 ```bash
@@ -264,13 +314,13 @@ Wait out the retrain ([Waiting for retrain](#waiting-for-retrain)). On the next
 
 **Rollback caveat:** Rollback restores the previous instructions but the model needs to retrain. Expect only **partial recovery** — prefer small-scope iterations (few fields at a time).
 
-**No regression:** Accept the iteration. Update `previous_metrics` and `previous_instructions` with the new values.
+**No regression:** Accept the iteration. Update `previous_metrics` (the complete payload again, not just F1) and `previous_instructions` with the new values.
 
 **Stopping criteria — stop the loop if:**
 
 - All fields meet the user's target F1 (default: 0.7)
 - Max iterations reached
-- No fields improved in the last 2 consecutive iterations (diminishing returns)
+- No fields improved by more than their own `regression_threshold` in the last 2 consecutive iterations (diminishing returns — a run of sub-threshold moves is not progress)
 
 ---
 
@@ -279,19 +329,23 @@ Wait out the retrain ([Waiting for retrain](#waiting-for-retrain)). On the next
 After the loop ends, print a summary:
 
 ```text
-Optimization complete after N iterations.
+Optimization complete after N iterations. Model version V1 -> V2.
 
-Field               | Baseline F1 | Final F1 | Change
---------------------|-------------|----------|-------
-Invoice Number      | 0.450       | 0.820    | +0.370
-Description         | 0.300       | 0.650    | +0.350
-Bill-To Name        | 0.900       | 0.900    | (unchanged)
-Vendor Address      | 0.600       | 0.400    | -0.200 (rolled back)
+Field           | Base F1 | Final F1 | Change    | Prec  | Rec   | Err   | Ann
+----------------|---------|----------|-----------|-------|-------|-------|----
+Invoice Number  | 0.450   | 0.820    | +0.370    | 0.850 | 0.790 | 0.200 |  40
+Description     | 0.300   | 0.650    | +0.350    | 0.700 | 0.610 | 0.395 |  38
+Bill-To Name    | 0.900   | 0.900    | unchanged | 0.900 | 0.900 | 0.100 |  40
+Vendor Address  | 0.600   | 0.400    | -0.200 (rolled back) | 0.410 | 0.390 | 0.600 | 40
+Freight Charge  | 1.000   | 0.889    | -0.111 (under its threshold, kept) | 1.000 | 0.800 | 0.200 | 5
 
-Overall project score: X.XX → Y.YY
+Project score:   X.XX (Quality) -> Y.YY (Quality)   ValidatedDocuments: D
 Iterations: N total, M with rollbacks
 Fields still below target (F1 < 0.7): [list]
+Fields whose regression_threshold sits above the flat 0.1 (too few Annotations to measure progress): [list, each tagged UPLOAD or REVIEW]
 Labelling gaps fixed: [list any fields re-labelled in 2a-check]
 ```
 
-If fields still need work, suggest the user run another round with more iterations.
+`ErrorRate` is the manual-correction burden left; `Annotations` tells a real plateau from an unmeasurable one.
+
+If fields still need work, suggest the user run another round with more iterations. For any field in the *too-few-`Annotations`* list, say plainly that its score cannot rise further until its sample grows, and which remedy grows it — **UPLOAD** (more documents) or **REVIEW** (the documents where it carries no label).

package/version-manifest.json

@@ -1,5 +1,5 @@
 {
   "schemaVersion": 2,
-  "skillsVersion": "1.201.0-preview.457",
+  "skillsVersion": "1.201.0-preview.488",
   "targetCli": "^1.201.0"
 }