@uipath/skills 1.195.0

package/skills/uipath-maestro-flow/references/evaluate/references/commands-reference.md

@@ -0,0 +1,306 @@
+# `uip maestro flow eval` Command Reference
+
+Complete syntax reference for every subcommand under `uip maestro flow eval`. All commands accept the global flags `--output <table|json|yaml|plain>` (default `json`), `--output-filter <jmespath>`, `--log-level <debug|info|warn|error>`, and `--log-file <path>`. Repeated reminders below: always pass `--output json` when an agent will parse the result.
+
+## Command Tree
+
+```
+uip maestro flow eval
+├── add        — Add a data point to an evaluation set
+├── list       — List data points in an evaluation set
+├── remove     — Remove a data point from an evaluation set
+├── set
+│   ├── add    — Create an evaluation set
+│   ├── list   — List evaluation sets
+│   └── remove — Remove an evaluation set
+├── evaluator
+│   ├── add    — Add an evaluator to the flow project
+│   ├── list   — List evaluators in a flow project
+│   └── remove — Remove an evaluator from a flow project
+├── simulation
+│   ├── add    — Add or update a simulation on a data point
+│   ├── list   — List simulations on a data point
+│   └── remove — Remove a simulation from a data point
+└── run
+    ├── start    — Start a Studio Web evaluation run
+    ├── status   — Check the status of a run
+    ├── results  — Get detailed per-data-point results
+    ├── list     — List runs for an eval set
+    └── compare  — Compare two runs side-by-side
+```
+
+`add` / `remove` / `list` at the top level operate on **data points** (test cases) inside an eval set. Data points are stored inline within the eval set JSON, not as separate files.
+
+## Common Options
+
+Every subcommand accepts:
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--path <path>` | No (defaults to `.`) | Flow project directory, or a solution directory containing exactly one Flow project |
+| `--output <fmt>` | No (default `json`) | `table`, `json`, `yaml`, or `plain` |
+| `--output-filter <expr>` | No | JMESPath expression applied to JSON output before printing |
+
+## Data Points (Test Cases)
+
+### `uip maestro flow eval add <name>`
+
+Add a data point to an eval set.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--set <name>` | Yes | Eval set name or ID |
+| `--inputs <json>` | No | Input values as a JSON object; keys must be declared as Flow input variables |
+| `--input-file <key=path>` | No | Attach a file as input `<key>`; **repeatable** |
+| `--expected <json>` | No | Expected output as a JSON object |
+| `--criteria <json>` | No | Per-evaluator criteria JSON object keyed by evaluator id |
+| `--search-text <text>` | No | Search text for `contains` evaluators |
+| `--path <path>` | No | (see Common Options) |
+
+Example:
+
+```bash
+uip maestro flow eval add greeting-test \
+  --set "Smoke Tests" \
+  --inputs '{"name":"Alice"}' \
+  --expected '{"greeting":"Hello, Alice!"}' \
+  --path ./MySolution/MyFlow --output json
+```
+
+### `uip maestro flow eval list`
+
+List data points in an eval set.
+
+```bash
+uip maestro flow eval list --set "Smoke Tests" --path ./MySolution/MyFlow --output json
+```
+
+### `uip maestro flow eval remove <id>`
+
+Remove a data point. `<id>` accepts the data point's UUID or its `name`.
+
+## Evaluation Sets
+
+### `uip maestro flow eval set add <name>`
+
+Create an evaluation set.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--evaluators <refs>` | No (default: all) | Comma-separated evaluator IDs or generated file base names; do not pass display names |
+| `--entry-point <id>` | No | Entry point node id stored as the eval set's `selectedEntrypoint` |
+| `--path <path>` | No | (see Common Options) |
+
+When `--evaluators` is omitted, the new eval set references **all** evaluators present in the project at creation time using their generated evaluator file refs. Prefer this when creating a set immediately after adding the evaluator(s). If passing `--evaluators`, use the generated id/file base returned by `evaluator add/list`, not the evaluator display name.
+
+### `uip maestro flow eval set list`
+
+List eval sets in the project.
+
+### `uip maestro flow eval set remove <id>`
+
+Remove an eval set. `<id>` accepts the eval set's UUID, `name`, or file base name.
+
+## Evaluators
+
+### `uip maestro flow eval evaluator add <name>`
+
+Create an evaluator file in the project's evaluators directory.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--type <type>` | Yes | One of: `exact-match`, `json-similarity`, `contains`, `llm-judge-output`, `llm-judge-strict-json`, `llm-judge-trajectory`, `llm-judge-trajectory-simulation` |
+| `--description <text>` | No | Evaluator description |
+| `--target-key <key>` | No | Output key the evaluator scores against (defaults to `*` — the entire output) |
+| `--model <model>` | No (Yes for llm-judge-*) | LLM model for LLM-judge evaluators (e.g. `gpt-4.1-2025-04-14`) |
+| `--prompt <prompt>` | No | Custom LLM judge prompt; defaults to a built-in template per type |
+| `--path <path>` | No | (see Common Options) |
+
+Only kebab-case `--type` values are accepted; PascalCase fails with an error.
+
+For LLM-judge evaluators, `--model` is effectively required — the cloud worker rejects an empty `model` before sending to the LLM gateway. See [evaluators-guide.md](evaluators-guide.md) for the seven types in detail.
+
+### `uip maestro flow eval evaluator list`
+
+List evaluators in the project.
+
+### `uip maestro flow eval evaluator remove <id>`
+
+Remove an evaluator. `<id>` accepts UUID, name, or file base name. Removing an evaluator does not auto-clean `evaluatorRefs` in eval sets — verify after removing.
+
+## Simulations
+
+Simulations intercept specific nodes (connectors, agents, sub-flows) during an eval run and replace their real execution with a controlled response. Each simulation targets a single component by its `componentId` and applies one of three strategies.
+
+### `uip maestro flow eval simulation add <component-id>`
+
+Add or replace a simulation on a data point. If a simulation for `<component-id>` already exists on the data point it is overwritten.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--set <name>` | Yes | Eval set name or ID |
+| `--data-point <id>` | Yes | Data point name or ID |
+| `--strategy <strategy>` | Yes | `Llm` or `Static` |
+| `--component-type <type>` | Yes | Component type (e.g. `connector`, `agent`, `subflow`) |
+| `--component-description <text>` | No | Human-readable label for the component |
+| `--simulation-instructions <text>` | No | LLM prompt describing what the component should return (for `Llm` strategy) |
+| `--mock-value <json>` | No | Static JSON output (for `Static` strategy) |
+| `--output-schema <json>` | No | JSON Schema describing the expected output shape; passed to the LLM to constrain its response. **Auto-resolved from the `.flow` file when omitted for `Llm` strategy** — fails if the node is not found or has no outputs. Pass explicitly to override. |
+| `--path <path>` | No | (see Common Options) |
+
+**Strategy guide:**
+
+| Strategy | When to use | Key flags |
+|----------|-------------|-----------|
+| `Llm` | Output should be realistic but non-deterministic | `--simulation-instructions`, `--output-schema` (auto-resolved) |
+| `Static` | Output is fixed and deterministic | `--mock-value` |
+
+**`--output-schema` auto-resolution for Llm simulations:** When omitted, the CLI reads the `.flow` file, finds the node by `<component-id>`, and derives the schema from the node's output definition (connector `outputJsonSchema`, agent `agentOutputVariables`, or `node.outputs`). Fails with an actionable error if the node is not found or has no outputs. Pass `--output-schema` explicitly to override. The JSON Schema is sent alongside `--simulation-instructions` to the LLM, telling it what shape the output must conform to. Without it the LLM generates free-form text. For a connector that returns `{ status, message }` you would pass:
+
+```bash
+--output-schema '{"type":"object","properties":{"status":{"type":"string"},"message":{"type":"string"}}}'
+```
+
+Example — LLM strategy:
+
+```bash
+uip maestro flow eval simulation add connector-send-email \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --strategy Llm \
+  --component-type connector \
+  --simulation-instructions "Pretend to send the email and return a success confirmation." \
+  --path ./MySolution/MyFlow --output json
+```
+
+Example — Static strategy:
+
+```bash
+uip maestro flow eval simulation add agent-lookup \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --strategy Static \
+  --component-type agent \
+  --mock-value '{"result": "found", "items": []}' \
+  --path ./MySolution/MyFlow --output json
+```
+
+### `uip maestro flow eval simulation list`
+
+List all simulations configured on a data point.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--set <name>` | Yes | Eval set name or ID |
+| `--data-point <id>` | Yes | Data point name or ID |
+| `--path <path>` | No | (see Common Options) |
+
+```bash
+uip maestro flow eval simulation list \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --path ./MySolution/MyFlow --output json
+```
+
+### `uip maestro flow eval simulation remove <component-id>`
+
+Remove a simulation from a data point. Returns an error if no simulation with the given `<component-id>` exists on the data point.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--set <name>` | Yes | Eval set name or ID |
+| `--data-point <id>` | Yes | Data point name or ID |
+| `--path <path>` | No | (see Common Options) |
+
+```bash
+uip maestro flow eval simulation remove connector-send-email \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --path ./MySolution/MyFlow --output json
+```
+
+## Run
+
+### `uip maestro flow eval run start`
+
+Start a Studio Web evaluation run. The Flow solution **must already exist in Studio Web** — see [upload-safety.md](upload-safety.md).
+
+| Flag | Required | Description | Default |
+|------|----------|-------------|---------|
+| `--set <name>` | Yes | Eval set name or ID | — |
+| `--solution-id <id>` | No | Solution ID from Studio Web | Auto-resolved from project metadata |
+| `--project-id <id>` | No | Flow project ID from Studio Web | Auto-resolved |
+| `--path <path>` | No | (see Common Options) | `.` |
+| `--entry-point <entry>` | No | Flow entry point path (e.g. `/Main.bpmn#start`) or start node ID | Eval set's `selectedEntrypoint` |
+| `--folder-key <key>` | No | Orchestrator folder key | Personal workspace |
+| `--debug-mode <mode>` | No | Studio Web debug mode override | (server default) |
+| `--wait` | No | Block until terminal state, then print results | `false` |
+| `--timeout <seconds>` | No | Max time to block on `--wait` | `600` (10 min) |
+
+Without `--wait`, returns immediately with `EvalSetRunId`. With `--wait`, the CLI polls until `Completed` or `Failed`, or `--timeout` elapses (the server-side run continues regardless).
+
+### `uip maestro flow eval run status <evalSetRunId>`
+
+Get current status. Terminal states: `Completed`, `Failed`.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--set <name>` | Yes | Eval set name or ID |
+| `--solution-id <id>` | No | Override solution ID |
+| `--project-id <id>` | No | Override project ID |
+| `--path <path>` | No | (see Common Options) |
+
+### `uip maestro flow eval run results <evalSetRunId>`
+
+Per-data-point results.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--set <name>` | Yes | Eval set name or ID |
+| `--only-failed` | No | Show only failed/errored data points |
+| `--verbose` | No | Include evaluator justifications |
+| `--export-format <json\|csv>` | No | Export results to a file |
+| `--solution-id`, `--project-id`, `--path` | No | (see start) |
+
+Per-row output fields: `DataPoint`, `Status`, `EvaluatorScores`, `Duration`, `Error` (plus `Justifications` when `--verbose`).
+
+### `uip maestro flow eval run list`
+
+List runs for an eval set.
+
+```bash
+uip maestro flow eval run list --set "Smoke Tests" --path ./MySolution/MyFlow --output json
+```
+
+### `uip maestro flow eval run compare <evalSetRunId>`
+
+Compare two runs side-by-side.
+
+| Flag | Required | Description |
+|------|----------|-------------|
+| `--compare-to <id>` | Yes | Second eval set run ID |
+| `--set <name>` | Yes | Eval set name or ID |
+| `--solution-id`, `--project-id`, `--path` | No | (see start) |
+
+`compare` aligns data points by `name` within the eval set. Comparing runs from different eval sets is meaningless.
+
+## Output Codes
+
+The CLI emits a `Code` field on every JSON response. Useful when filtering or scripting:
+
+| Subcommand | `Code` |
+|------------|--------|
+| `eval add` | `FlowEvalAdd` |
+| `eval list` | `FlowEvalList` |
+| `eval remove` | `FlowEvalRemove` |
+| `eval set add` / `list` / `remove` | `FlowEvalSetAdd` / `FlowEvalSetList` / `FlowEvalSetRemove` |
+| `eval evaluator add` / `list` / `remove` | `FlowEvalEvaluatorAdd` / `FlowEvalEvaluatorList` / `FlowEvalEvaluatorRemove` |
+| `eval simulation add` / `list` / `remove` | `FlowEvalSimulationAdd` / `FlowEvalSimulationList` / `FlowEvalSimulationRemove` |
+| `eval run start` (no `--wait`) | `MaestroFlowEvalRunStarted` |
+| `eval run start --wait` (summary) | `MaestroFlowEvalRunCompleted` |
+| `eval run status` | `MaestroFlowEvalRunStatus` |
+| `eval run results` | `MaestroFlowEvalRunResults` |
+| `eval run list` | `MaestroFlowEvalRunList` |
+| `eval run compare` | `MaestroFlowEvalRunComparison` |
+
+If actual emitted codes diverge from the table above, trust the JSON output — file an issue.

package/skills/uipath-maestro-flow/references/evaluate/references/eval-sets-guide.md

@@ -0,0 +1,226 @@
+# Eval Sets and Data Points
+
+An eval set is a JSON file that bundles test cases (data points) with the evaluators that score them. Each Flow project can have multiple eval sets — typically one per scenario (smoke, regression, edge cases).
+
+## Eval Set Lifecycle
+
+```bash
+# Create
+uip maestro flow eval set add "<set_name>" \
+  [--evaluators <ref1>,<ref2>] \
+  [--entry-point <node_id>] \
+  --path <flow_project> --output json
+
+# List
+uip maestro flow eval set list --path <flow_project> --output json
+
+# Remove
+uip maestro flow eval set remove "<id_or_name>" --path <flow_project> --output json
+```
+
+### `--evaluators`
+
+Comma-separated list of evaluator IDs or generated file base names. When omitted, the new eval set links to **all evaluators** present in the project at creation time and writes the generated evaluator file refs (for example, `greeting-match-62c0793a.json`). Prefer omission when the eval set should use every current evaluator.
+
+Do **not** pass an evaluator display name such as `greeting-match` to `--evaluators`; that writes the display name into `evaluatorRefs`, and Studio Web eval runs will not resolve it. If you need an explicit list, get the generated id/file ref from `uip maestro flow eval evaluator add/list --output json` and pass that value instead.
+
+Subsequent evaluators added to the project do NOT auto-attach — link them by re-running `set add` with explicit generated refs, or carefully repair the eval set's `evaluatorRefs` array using the evaluator file refs the CLI created.
+
+### `--entry-point`
+
+Stored as `selectedEntrypoint` on the eval set. Accepts either:
+
+- A flow path: `/Main.bpmn#start`
+- A start node ID: `start_a1b2c3d4`
+
+The eval set's entry point is used by `eval run start` unless `--entry-point` is overridden at run time. Pin one explicitly when the flow has multiple entry points; otherwise the first start node found is used and may not be the one you intend.
+
+## Data Point Lifecycle
+
+Data points (test cases) live **inline** inside the eval set JSON, not as separate files. CRUD via `uip maestro flow eval add/list/remove`:
+
+```bash
+# Add a data point
+uip maestro flow eval add "<data_point_name>" \
+  --set "<set_name>" \
+  --inputs '{"...":"..."}' \
+  [--expected '{"...":"..."}'] \
+  [--criteria '{"<evaluator_id>": {...}}'] \
+  [--input-file <key>=<path>] \
+  [--search-text "<text>"] \
+  --path <flow_project> --output json
+
+# List
+uip maestro flow eval list --set "<set_name>" --path <flow_project> --output json
+
+# Remove
+uip maestro flow eval remove "<id_or_name>" --set "<set_name>" --path <flow_project> --output json
+```
+
+### `--inputs`
+
+JSON object whose keys must match the flow's declared input variables for the chosen entry point. `eval add` validates these keys and fails fast when an input key is not declared in the `.flow` file. Add missing variables first, or change the data point input JSON to match the Flow schema.
+
+For a simple string input named `name`:
+
+```bash
+uip maestro flow variable add ./MySolution/MyFlow/MyFlow.flow name \
+  --direction in --type string --output json
+```
+
+### `--expected`
+
+JSON object representing the expected output. How it is consumed depends on the evaluator type:
+
+| Evaluator type | Reads `--expected` how |
+|----------------|-----------------------|
+| `exact-match` | Compares the output against `expected` verbatim (or `expected[targetKey]` if `--target-key` is set on the evaluator) |
+| `json-similarity` | Tree-compared with tolerance |
+| `contains` | Not used directly — see `--search-text` |
+| `llm-judge-output` | Substituted into `{{ExpectedOutput}}` |
+| `llm-judge-strict-json` | Substituted per-key into `{{ExpectedOutput}}` |
+| `llm-judge-trajectory` | Used along with `--expected-agent-behavior` (set via `--criteria`) |
+| `llm-judge-trajectory-simulation` | Same as `trajectory` plus simulation context |
+
+### `--criteria`
+
+Per-evaluator overrides keyed by evaluator ID. Use this when one eval set contains multiple evaluators and you want each to score against different criteria for the same data point. Most common shape:
+
+```json
+{
+  "<evaluator_ref_for_trajectory>": {
+    "expectedAgentBehavior": "Agent calls the weather tool with the user's location, then returns a one-sentence summary."
+  },
+  "<evaluator_ref_for_output>": {
+    "expectedOutput": {"summary": "Sunny in NYC, 72°F"}
+  }
+}
+```
+
+If `--criteria` is omitted, evaluators fall back to `--expected` (for output evaluators) and to defaults (for trajectory evaluators). Always pass an explicit `expectedAgentBehavior` for trajectory evaluators — the LLM's score against an empty `{{ExpectedAgentBehavior}}` is meaningless.
+
+### `--input-file <key>=<path>`
+
+Repeatable. Attach a file as input under the given key. The CLI stages the file alongside the eval set so the runtime can read it during evaluation. Useful when the flow accepts file inputs (PDFs, CSVs, images) rather than JSON only.
+
+```bash
+uip maestro flow eval add invoice-test \
+  --set "Smoke Tests" \
+  --inputs '{"orderId":"ORD-123"}' \
+  --input-file invoice=./fixtures/invoice-123.pdf \
+  --input-file receipt=./fixtures/receipt-123.png \
+  --expected '{"total": 142.50}' \
+  --path ./MySolution/MyFlow --output json
+```
+
+The file content is referenced from the eval set; do not delete the source file before the run completes.
+
+### `--search-text`
+
+Convenience flag for `contains` evaluators only. Attaches the search string to the data point so the contains evaluator knows what substring to test for. Equivalent to writing `criteria` for the contains evaluator.
+
+```bash
+uip maestro flow eval add success-keyword \
+  --set "Smoke Tests" \
+  --inputs '{"task":"deploy"}' \
+  --search-text "deployment succeeded" \
+  --path ./MySolution/MyFlow --output json
+```
+
+## Eval Set JSON Shape
+
+A typical eval set JSON file (created by `set add`):
+
+```json
+{
+  "id": "<uuid>",
+  "name": "Smoke Tests",
+  "version": "1.0",
+  "evaluatorRefs": ["<evaluator-file-ref-1>.json", "<evaluator-file-ref-2>.json"],
+  "selectedEntrypoint": "/Main.bpmn#start",
+  "evaluations": [
+    {
+      "id": "<uuid>",
+      "name": "hello-test",
+      "inputs": {"message": "hello"},
+      "expectedOutput": {"reply": "Hello! How can I help you?"},
+      "evaluationCriterias": {
+        "<evaluator-ref-2>": {
+          "expectedAgentBehavior": "Agent responds with a friendly greeting."
+        }
+      }
+    }
+  ]
+}
+```
+
+Key fields:
+
+- `version: "1.0"` distinguishes the new eval format from any legacy variants. Keep this. <!-- version-check-skip --> (eval-set file schema version, CLI-emitted by `eval set add` — not a `.flow` version)
+- `evaluatorRefs` references evaluator files generated by the CLI. Prefer CLI commands over hand-editing these refs.
+- `selectedEntrypoint` is the eval set's pinned entry point. Override per-run with `eval run start --entry-point`.
+- `evaluations[]` is the inline list of data points. Order is informational, not significant.
+- `evaluationCriterias` (note plural) is the per-data-point per-evaluator override map.
+
+## Aligning Inputs with the Flow Schema
+
+The data point's `inputs` JSON must match the flow's input schema at the chosen entry point. Mismatches produce `eval add` errors such as `Input "name" is not declared as an input variable in the flow`. Two ways to verify before adding data points:
+
+1. Inspect `<flow>.flow` in the project for `variables` entries with `direction: "in"`.
+2. Run `uip maestro flow variable list <flow_file> --output json`.
+
+## Simulations on Data Points
+
+Simulations replace selected nodes during an eval run with controlled outputs, so the flow is evaluated without calling real external services. Each simulation targets one component by its node `id` from the `.flow` file.
+
+```bash
+# Add an LLM simulation (non-deterministic, prompt-guided)
+uip maestro flow eval simulation add <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Llm \
+  --component-type connector \
+  --simulation-instructions "Pretend to send the email and return success." \
+  --output-schema '{"type":"object","properties":{"status":{"type":"string"},"messageId":{"type":"string"}}}' \
+  --path <flow_project> --output json
+
+# Add a Static simulation (fixed JSON output)
+uip maestro flow eval simulation add <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Static \
+  --component-type agent \
+  --mock-value '{"status":"ok"}' \
+  --path <flow_project> --output json
+
+# List simulations on a data point
+uip maestro flow eval simulation list \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+
+# Remove a simulation
+uip maestro flow eval simulation remove <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+```
+
+**Strategy selection:**
+
+| Strategy | Use when | Key flags |
+|----------|----------|-----------|
+| `Llm` | Output should be plausible but non-deterministic | `--simulation-instructions`, `--output-schema` (auto-resolved) |
+| `Static` | Output must be identical every run | `--mock-value <json>` |
+
+For `Llm` simulations, `--output-schema` is **auto-resolved from the `.flow` file** — the CLI reads the node's output definition (connector `outputJsonSchema`, agent `agentOutputVariables`, or `node.outputs`) and injects it automatically. Pass `--output-schema` explicitly only when you need to override the derived schema or the node has no declared outputs.
+
+Simulations are stored inline in the data point's `simulations` array within the eval set JSON. Running `simulation add` twice for the same `<component-id>` on the same data point replaces the existing simulation.
+
+## Anti-patterns
+
+- **Don't hand-write `id` UUIDs on data points.** Use `uip maestro flow eval add` so the CLI generates fresh UUIDs and keeps `evalSetId` consistent with the parent set.
+- **Don't pass `--inputs` keys that aren't in the flow's input schema.** The CLI rejects unknown keys before writing the data point.
+- **Don't set `--expected '{}'` and skip `--criteria` for trajectory evaluators.** Both placeholders end up empty; scoring is meaningless.
+- **Don't delete attached input files before the run completes.** The CLI references them by path until upload to Studio Web finishes.
+- **Don't expect `--evaluators` on `set add` to auto-update.** Adding new evaluators later does not retroactively link them to existing eval sets.

package/skills/uipath-maestro-flow/references/evaluate/references/evaluators-guide.md

@@ -0,0 +1,143 @@
+# Flow Evaluators Guide
+
+The seven evaluator types for `uip maestro flow eval evaluator add --type <type>`. Flow uses the same `uipath-*` evaluator type IDs as the coded evaluator engine (NOT the legacy IDs the low-code `uipath-agents` skill uses). Each `--type` kebab-case value maps to a `uipath-*` evaluator under the hood:
+
+| `--type` flag | Internal `evaluatorTypeId` | LLM-based | Score scale |
+|---------------|---------------------------|-----------|-------------|
+| `exact-match` | `uipath-exact-match` | No | Binary (0/1) |
+| `json-similarity` | `uipath-json-similarity` | No | Continuous (0–1) |
+| `contains` | `uipath-contains` | No | Binary (0/1) |
+| `llm-judge-output` | `uipath-llm-judge-output-semantic-similarity` | Yes | Continuous (0–1) |
+| `llm-judge-strict-json` | `uipath-llm-judge-output-strict-json-similarity` | Yes | Continuous (0–1) |
+| `llm-judge-trajectory` | `uipath-llm-judge-trajectory-similarity` | Yes | Continuous (0–1) |
+| `llm-judge-trajectory-simulation` | `uipath-llm-judge-trajectory-simulation` | Yes | Continuous (0–1) |
+
+The four classification and tool-call evaluators (`uipath-binary-classification`, `uipath-multiclass-classification`, `uipath-tool-call-*`) that exist in the coded SDK are **not** exposed by `uip maestro flow eval evaluator add`. Use the seven listed above.
+
+## When to Pick Each Type
+
+| Goal | Type | Notes |
+|------|------|-------|
+| Strict equality of a known output value | `exact-match` | Binary; no LLM. Cheapest. Use for deterministic agents. |
+| Compare structured JSON outputs with tolerance for ordering / minor diffs | `json-similarity` | Tree-based; numeric tolerance ~1%, string Levenshtein. |
+| Pass/fail on whether a substring appears | `contains` | Use `--search-text` on the data point, not the evaluator. |
+| Score natural-language similarity between expected and actual | `llm-judge-output` | Default LLM judge. Set `--model`. |
+| Per-key strict JSON scoring with LLM judgment on differences | `llm-judge-strict-json` | LLM penalizes per-field. Use when each output key matters independently. |
+| Score whether the agent's reasoning path matches expected behavior | `llm-judge-trajectory` | Reads agent run history. Use `--expected-agent-behavior` on data points. |
+| As above but with simulated tool calls / inputs | `llm-judge-trajectory-simulation` | Use when the flow can't safely execute real tools during eval. |
+
+## Adding an Evaluator (CLI)
+
+```bash
+# Deterministic — no model needed
+uip maestro flow eval evaluator add exact-greeting \
+  --type exact-match \
+  --target-key "greeting" \
+  --path ./MySolution/MyFlow --output json
+
+# LLM-judge — model is effectively required
+uip maestro flow eval evaluator add greeting-quality \
+  --type llm-judge-output \
+  --model gpt-4.1-2025-04-14 \
+  --description "Score greeting tone and completeness" \
+  --path ./MySolution/MyFlow --output json
+```
+
+Critical: pass `--model` on every `llm-judge-*` evaluator. Empty `model` triggers a 500 from the LLM gateway only after retries — unfriendly to debug.
+
+## `--target-key`
+
+Defaults to `*` (the entire output object). Set a specific key when only one field of the output should be scored:
+
+- Output: `{"greeting": "Hello!", "metadata": {"latencyMs": 12}}` 
+- `--target-key "greeting"` scores only the greeting string, ignoring metadata noise.
+- `--target-key "*"` (default) scores the whole object.
+
+## Evaluator JSON Shape
+
+The CLI writes evaluator JSON files into the project's evaluator directory. Filenames follow `<name>-<suffix>.json` for CLI-created files; hand-written files can use any descriptive name. Let `eval set add` write `evaluatorRefs` instead of hand-editing them.
+
+### Deterministic example (`exact-match`)
+
+```json
+{
+  "id": "<generate-uuid>",
+  "name": "exact-greeting",
+  "description": "Strict greeting match",
+  "version": "1.0",
+  "evaluatorTypeId": "uipath-exact-match",
+  "evaluatorConfig": {
+    "name": "exact-greeting",
+    "targetOutputKey": "greeting",
+    "caseSensitive": false,
+    "negated": false
+  }
+}
+```
+
+`exact-match`, `json-similarity`, and `contains` do NOT carry a `model` or `prompt`.
+
+### LLM-judge example (`llm-judge-output`)
+
+```json
+{
+  "id": "<generate-uuid>",
+  "name": "greeting-quality",
+  "description": "Score greeting tone and completeness",
+  "version": "1.0",
+  "evaluatorTypeId": "uipath-llm-judge-output-semantic-similarity",
+  "evaluatorConfig": {
+    "name": "greeting-quality",
+    "model": "gpt-4.1-2025-04-14",
+    "temperature": 0,
+    "maxTokens": 4096,
+    "targetOutputKey": "*",
+    "prompt": "Score 0-1 the semantic similarity of {{ActualOutput}} vs {{ExpectedOutput}}. Return JSON {\"score\": N, \"reason\": \"...\"}."
+  }
+}
+```
+
+Template variables per type:
+
+| Type | Placeholders |
+|------|-------------|
+| `llm-judge-output` | `{{ExpectedOutput}}`, `{{ActualOutput}}` |
+| `llm-judge-strict-json` | `{{ExpectedOutput}}`, `{{ActualOutput}}` |
+| `llm-judge-trajectory` | `{{AgentRunHistory}}`, `{{ExpectedAgentBehavior}}`, `{{UserOrSyntheticInput}}` |
+| `llm-judge-trajectory-simulation` | `{{AgentRunHistory}}`, `{{ExpectedAgentBehavior}}`, `{{UserOrSyntheticInput}}`, `{{SimulationInstructions}}` |
+
+When `--prompt` is omitted, the CLI inserts a built-in default for each type. Override with `--prompt` only when you need different wording or scoring criteria — keep the same placeholders.
+
+## Custom Prompts
+
+```bash
+uip maestro flow eval evaluator add strict-match \
+  --type llm-judge-output \
+  --model gpt-4.1-2025-04-14 \
+  --prompt 'Score 0-1 how closely {{ActualOutput}} matches {{ExpectedOutput}}. Return JSON {"score": N, "reason": "..."}.' \
+  --path ./MySolution/MyFlow --output json
+```
+
+Unknown placeholders are passed through to the LLM as literal text — they are not silent errors but they will not be substituted with run data.
+
+## Removing an Evaluator
+
+```bash
+uip maestro flow eval evaluator remove greeting-quality \
+  --path ./MySolution/MyFlow --output json
+```
+
+Removing an evaluator does NOT auto-clean `evaluatorRefs` arrays in eval sets that reference it. After removing, re-list eval sets and reconcile any stale refs:
+
+```bash
+uip maestro flow eval set list --path ./MySolution/MyFlow --output json
+```
+
+## Anti-patterns
+
+- **Don't pass `--type` in PascalCase.** `ExactMatch` fails; only `exact-match` is accepted.
+- **Don't omit `--model` on `llm-judge-*`.** The cloud worker fail-fasts before calling the LLM gateway; the resulting 500 is unhelpful.
+- **Don't hand-write evaluator refs unless you are repairing an eval set.** Omit `--evaluators` so the CLI links all current evaluators, or use `eval set add --evaluators <generated_id_or_file_base>`. Do not pass the evaluator display name.
+- **Don't copy evaluator JSON across projects without regenerating UUIDs.** `id` collisions silently corrupt resolution.
+- **Don't use deterministic evaluators (`exact-match`, `json-similarity`, `contains`) for natural-language outputs.** They will fail almost every test that paraphrases.
+- **Don't mix trajectory and output evaluators on the same data point unless the data point provides both `--expected` and `--expected-agent-behavior`.** Otherwise one of them scores against an empty placeholder.