@uipath/skills 1.204.0 → 1.204.1

package/skills/uipath-maestro-flow/references/evaluate/eval-sets-guide.md

@@ -0,0 +1,227 @@
+# Eval Sets and Data Points
+
+An eval set is a JSON file containing test cases (data points) and evaluators. A Flow project may have multiple sets, typically one per scenario.
+
+## Eval Set Lifecycle
+
+Run:
+
+```bash
+uip maestro flow eval set add "<set_name>" \
+  [--evaluators <ref1>,<ref2>] \
+  [--entry-point <node_id>] \
+  --path <flow_project> --output json
+
+uip maestro flow eval set list --path <flow_project> --output json
+
+uip maestro flow eval set remove "<id_or_name>" --path <flow_project> --output json
+```
+
+`--evaluators` accepts comma-separated evaluator IDs or generated file base names. If omitted, the set links to all evaluators present at creation and writes generated evaluator file refs such as `greeting-match-62c0793a.json`; prefer omission when using every current evaluator. Do not pass an evaluator display name such as `greeting-match`: this writes the display name into `evaluatorRefs`, which Studio Web eval runs cannot resolve. For an explicit list, run `uip maestro flow eval evaluator add/list --output json` and pass the generated ID/file ref. Evaluators added later are not auto-attached; rerun `set add` with explicit generated refs or carefully repair `evaluatorRefs` using CLI-created evaluator file refs.
+
+`--entry-point` is stored as `selectedEntrypoint` and accepts `/Main.bpmn#start` or a start node ID such as `start_a1b2c3d4`. `eval run start` uses it unless overridden with `--entry-point`. Pin it when multiple entry points exist; otherwise the first start node found is used.
+
+## Data Point Lifecycle
+
+Data points live inline in the eval set JSON. Run:
+
+```bash
+uip maestro flow eval add "<data_point_name>" \
+  --set "<set_name>" \
+  --inputs '{"...":"..."}' \
+  [--expected '{"...":"..."}'] \
+  [--criteria '{"<evaluator_id>": {...}}'] \
+  [--input-file <key>=<path>] \
+  [--search-text "<text>"] \
+  --path <flow_project> --output json
+
+uip maestro flow eval list --set "<set_name>" --path <flow_project> --output json
+
+uip maestro flow eval remove "<id_or_name>" --set "<set_name>" --path <flow_project> --output json
+```
+
+<a id="--criteria"></a>
+<a id="--input-file-keypath"></a>
+### Inputs, Expected Values, and Criteria
+
+`--inputs` must contain only keys matching the chosen entry point's declared input variables. If a variable is missing, add it or change the input JSON. To add a string input named `name`, run:
+
+```bash
+uip maestro flow variable add ./MySolution/MyFlow/MyFlow.flow name \
+  --direction in --type string --output json
+```
+
+Evaluator behavior:
+
+- `exact-match`: compares output verbatim, or `expected[targetKey]` when the evaluator has `--target-key`.
+- `json-similarity`: tree-compares with tolerance.
+- `contains`: does not use `--expected`; use `--search-text`.
+- `llm-judge-output`: substitutes `--expected` into `{{ExpectedOutput}}`.
+- `llm-judge-strict-json`: substitutes it per key into `{{ExpectedOutput}}`.
+- `llm-judge-trajectory`: uses it with `--expected-agent-behavior` set through `--criteria`.
+- `llm-judge-trajectory-simulation`: behaves like `trajectory`, with simulation context.
+
+`--criteria` contains per-evaluator overrides keyed by evaluator ID. Use it when evaluators in one set need different criteria:
+
+```json
+{
+  "<evaluator_ref_for_trajectory>": {
+    "expectedAgentBehavior": "Agent calls the weather tool with the user's location, then returns a one-sentence summary."
+  },
+  "<evaluator_ref_for_output>": {
+    "expectedOutput": {"summary": "Sunny in NYC, 72°F"}
+  }
+}
+```
+
+When omitted, output evaluators fall back to `--expected` and trajectory evaluators use defaults. Always provide `expectedAgentBehavior` for trajectory evaluators; scoring against an empty `{{ExpectedAgentBehavior}}` is meaningless.
+
+### File and Contains Inputs
+
+`--input-file <key>=<path>` is repeatable and attaches a staged file under the specified key for runtime use, including PDFs, CSVs, and images. Do not delete the source before the run completes.
+
+`--search-text` is for `contains` evaluators. It attaches the substring to test and is equivalent to writing `criteria` for that evaluator.
+
+## Eval Set JSON Shape
+
+```json
+{
+  "id": "<uuid>",
+  "name": "Smoke Tests",
+  "version": "1.0",
+  "evaluatorRefs": ["<evaluator-file-ref-1>.json", "<evaluator-file-ref-2>.json"],
+  "selectedEntrypoint": "/Main.bpmn#start",
+  "evaluations": [
+    {
+      "id": "<uuid>",
+      "name": "hello-test",
+      "inputs": {"message": "hello"},
+      "expectedOutput": {"reply": "Hello! How can I help you?"},
+      "evaluationCriterias": {
+        "<evaluator-ref-2>": {
+          "expectedAgentBehavior": "Agent responds with a friendly greeting."
+        }
+      }
+    }
+  ]
+}
+```
+
+Keep `version: "1.0"`; it identifies the new eval format. <!-- version-check-skip --> (eval-set file schema version, CLI-emitted by `eval set add` — not a `.flow` version)
+
+- `evaluatorRefs` references CLI-generated evaluator files; prefer CLI commands over hand-editing.
+- `selectedEntrypoint` pins the set's entry point; override it per run with `eval run start --entry-point`.
+- `evaluations[]` is the inline data-point list; order is informational.
+- `evaluationCriterias` (plural) maps per-data-point, per-evaluator overrides.
+
+## Aligning Inputs with the Flow Schema
+
+The data point's `inputs` must match the chosen entry point's input schema. Mismatches produce errors such as `Input "name" is not declared as an input variable in the flow`. Before adding data points, inspect `<flow>.flow` for `variables` entries with `direction: "in"`, or run:
+
+```bash
+uip maestro flow variable list <flow_file> --output json
+```
+
+## Simulations on Data Points
+
+Simulations replace selected nodes during an eval run with controlled outputs. Target a component by its `.flow` node `id`. Run:
+
+```bash
+uip maestro flow eval simulation add <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Llm \
+  --component-type connector \
+  --simulation-instructions "Pretend to send the email and return success." \
+  --path <flow_project> --output json
+
+uip maestro flow eval simulation add <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Static \
+  --component-type agent \
+  --mock-value '{"status":"ok"}' \
+  --path <flow_project> --output json
+
+uip maestro flow eval simulation list \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+
+uip maestro flow eval simulation remove <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+```
+
+| Strategy | Use | Required or relevant flags |
+|---|---|---|
+| `Llm` | Plausible, non-deterministic output | `--simulation-instructions` (output schema auto-resolved) |
+| `Static` | Identical output every run | `--mock-value <json>` |
+
+The output schema is always auto-resolved — for both top-level and child (`--parent`) simulations. Top-level reads the `.flow` node outputs; child simulations resolve from the `.flow` edges (inline agents), `agent.json` resources (same-solution agents), or the platform API (published agents, requires `uip login`).
+
+Simulations are stored inline in the data point's `simulations` array. Adding one for an existing `<component-id>` and data point replaces the existing simulation.
+
+### Child Simulations (Agent Tool Simulation)
+
+When a flow has an agent node, you can simulate individual tools inside that agent instead of mocking the whole agent as one unit. Use `--parent <agent-component-id>` to target a child tool. The child's `<component-id>` is the tool's runtime name (e.g. `Web_Search`, `Send_Email`), not a workflow node ID.
+
+```bash
+# Add a child tool simulation (Static — fixed output).
+# If no parent simulation exists for <agent-node-id>, one is auto-created
+# (type: agent, strategy: Llm).
+uip maestro flow eval simulation add Web_Search \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Static \
+  --mock-value '{"results": [{"title": "Example", "url": "https://example.com"}]}' \
+  --path <flow_project> --output json
+
+# Add a child tool simulation (Llm — prompt-guided).
+# Output schema is auto-resolved from the agent's tool definitions
+# (inline, same-solution, or published agents).
+uip maestro flow eval simulation add Send_Email \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Llm \
+  --simulation-instructions "Return a success status with a generated messageId." \
+  --path <flow_project> --output json
+
+# List child simulations on an agent node
+uip maestro flow eval simulation list \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+
+# Remove a child simulation
+uip maestro flow eval simulation remove Web_Search \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+```
+
+No separate parent simulation step is needed — passing `--parent` auto-creates the parent simulation (type `agent`, strategy `Llm`) if it does not exist yet. This means a single command is enough to add a child tool simulation.
+
+When `--parent` is used, `--component-type` defaults to `Node` (the convention for child tool simulations). You can override it if needed.
+
+The output schema is auto-resolved for child simulations on all agent types:
+- **Inline canvas agents** (`uipath.agent.*`): resolved from the child tool node's outputs in the `.flow` file.
+- **Same-solution agents** (with `inputs.source`): resolved from the inline agent's `agent.json` resources.
+- **Published agents** (`uipath.core.agent.*`): resolved via the platform API (`simulatableComponents`). Requires `uip login`.
+
+For `Static` strategy, the CLI also validates that `--mock-value` keys match the resolved schema properties, catching shape mismatches before the eval run.
+
+Child simulations are stored in the parent's `childSimulations` array in the eval set JSON. Running `simulation add` with `--parent` twice for the same child `<component-id>` replaces the existing child simulation.
+
+## Anti-patterns
+
+- Do not hand-write data-point `id` UUIDs. Run `uip maestro flow eval add`; the CLI generates fresh UUIDs and maintains `evalSetId`.
+- Do not pass `--inputs` keys absent from the flow input schema; the CLI rejects them.
+- Do not set `--expected '{}'` while omitting `--criteria` for trajectory evaluators; both placeholders are empty and scoring is meaningless.
+- Do not delete attached input files before the run completes; the CLI references them until upload to Studio Web finishes.
+- Do not expect `--evaluators` on `set add` to update automatically; later evaluators are not retroactively linked.

package/skills/uipath-maestro-flow/references/evaluate/evaluators-guide.md

@@ -0,0 +1,143 @@
+# Flow Evaluators Guide
+
+The seven evaluator types for `uip maestro flow eval evaluator add --type <type>`. Flow uses the same `uipath-*` evaluator type IDs as the coded evaluator engine (NOT the legacy IDs the low-code `uipath-agents` skill uses). Each `--type` kebab-case value maps to a `uipath-*` evaluator under the hood:
+
+| `--type` flag | Internal `evaluatorTypeId` | LLM-based | Score scale |
+|---------------|---------------------------|-----------|-------------|
+| `exact-match` | `uipath-exact-match` | No | Binary (0/1) |
+| `json-similarity` | `uipath-json-similarity` | No | Continuous (0–1) |
+| `contains` | `uipath-contains` | No | Binary (0/1) |
+| `llm-judge-output` | `uipath-llm-judge-output-semantic-similarity` | Yes | Continuous (0–1) |
+| `llm-judge-strict-json` | `uipath-llm-judge-output-strict-json-similarity` | Yes | Continuous (0–1) |
+| `llm-judge-trajectory` | `uipath-llm-judge-trajectory-similarity` | Yes | Continuous (0–1) |
+| `llm-judge-trajectory-simulation` | `uipath-llm-judge-trajectory-simulation` | Yes | Continuous (0–1) |
+
+The four classification and tool-call evaluators (`uipath-binary-classification`, `uipath-multiclass-classification`, `uipath-tool-call-*`) that exist in the coded SDK are **not** exposed by `uip maestro flow eval evaluator add`. Use the seven listed above.
+
+## When to Pick Each Type
+
+| Goal | Type | Notes |
+|------|------|-------|
+| Strict equality of a known output value | `exact-match` | Binary; no LLM. Cheapest. Use for deterministic agents. |
+| Compare structured JSON outputs with tolerance for ordering / minor diffs | `json-similarity` | Tree-based; numeric tolerance ~1%, string Levenshtein. |
+| Pass/fail on whether a substring appears | `contains` | Use `--search-text` on the data point, not the evaluator. |
+| Score natural-language similarity between expected and actual | `llm-judge-output` | Default LLM judge. Set `--model`. |
+| Per-key strict JSON scoring with LLM judgment on differences | `llm-judge-strict-json` | LLM penalizes per-field. Use when each output key matters independently. |
+| Score whether the agent's reasoning path matches expected behavior | `llm-judge-trajectory` | Reads agent run history. Use `--expected-agent-behavior` on data points. |
+| As above but with simulated tool calls / inputs | `llm-judge-trajectory-simulation` | Use when the flow can't safely execute real tools during eval. |
+
+## Adding an Evaluator (CLI)
+
+```bash
+# Deterministic — no model needed
+uip maestro flow eval evaluator add exact-greeting \
+  --type exact-match \
+  --target-key "greeting" \
+  --path ./MySolution/MyFlow --output json
+
+# LLM-judge — model is effectively required
+uip maestro flow eval evaluator add greeting-quality \
+  --type llm-judge-output \
+  --model gpt-4.1-2025-04-14 \
+  --description "Score greeting tone and completeness" \
+  --path ./MySolution/MyFlow --output json
+```
+
+Critical: pass `--model` on every `llm-judge-*` evaluator. Empty `model` triggers a 500 from the LLM gateway only after retries — unfriendly to debug.
+
+## `--target-key`
+
+Defaults to `*` (the entire output object). Set a specific key when only one field of the output should be scored:
+
+- Output: `{"greeting": "Hello!", "metadata": {"latencyMs": 12}}` 
+- `--target-key "greeting"` scores only the greeting string, ignoring metadata noise.
+- `--target-key "*"` (default) scores the whole object.
+
+## Evaluator JSON Shape
+
+The CLI writes evaluator JSON files into the project's evaluator directory. Filenames follow `<name>-<suffix>.json` for CLI-created files; hand-written files can use any descriptive name. Let `eval set add` write `evaluatorRefs` instead of hand-editing them.
+
+### Deterministic example (`exact-match`)
+
+```json
+{
+  "id": "<generate-uuid>",
+  "name": "exact-greeting",
+  "description": "Strict greeting match",
+  "version": "1.0",
+  "evaluatorTypeId": "uipath-exact-match",
+  "evaluatorConfig": {
+    "name": "exact-greeting",
+    "targetOutputKey": "greeting",
+    "caseSensitive": false,
+    "negated": false
+  }
+}
+```
+
+`exact-match`, `json-similarity`, and `contains` do NOT carry a `model` or `prompt`.
+
+### LLM-judge example (`llm-judge-output`)
+
+```json
+{
+  "id": "<generate-uuid>",
+  "name": "greeting-quality",
+  "description": "Score greeting tone and completeness",
+  "version": "1.0",
+  "evaluatorTypeId": "uipath-llm-judge-output-semantic-similarity",
+  "evaluatorConfig": {
+    "name": "greeting-quality",
+    "model": "gpt-4.1-2025-04-14",
+    "temperature": 0,
+    "maxTokens": 4096,
+    "targetOutputKey": "*",
+    "prompt": "Score 0-1 the semantic similarity of {{ActualOutput}} vs {{ExpectedOutput}}. Return JSON {\"score\": N, \"reason\": \"...\"}."
+  }
+}
+```
+
+Template variables per type:
+
+| Type | Placeholders |
+|------|-------------|
+| `llm-judge-output` | `{{ExpectedOutput}}`, `{{ActualOutput}}` |
+| `llm-judge-strict-json` | `{{ExpectedOutput}}`, `{{ActualOutput}}` |
+| `llm-judge-trajectory` | `{{AgentRunHistory}}`, `{{ExpectedAgentBehavior}}`, `{{UserOrSyntheticInput}}` |
+| `llm-judge-trajectory-simulation` | `{{AgentRunHistory}}`, `{{ExpectedAgentBehavior}}`, `{{UserOrSyntheticInput}}`, `{{SimulationInstructions}}` |
+
+When `--prompt` is omitted, the CLI inserts a built-in default for each type. Override with `--prompt` only when you need different wording or scoring criteria — keep the same placeholders.
+
+## Custom Prompts
+
+```bash
+uip maestro flow eval evaluator add strict-match \
+  --type llm-judge-output \
+  --model gpt-4.1-2025-04-14 \
+  --prompt 'Score 0-1 how closely {{ActualOutput}} matches {{ExpectedOutput}}. Return JSON {"score": N, "reason": "..."}.' \
+  --path ./MySolution/MyFlow --output json
+```
+
+Unknown placeholders are passed through to the LLM as literal text — they are not silent errors but they will not be substituted with run data.
+
+## Removing an Evaluator
+
+```bash
+uip maestro flow eval evaluator remove greeting-quality \
+  --path ./MySolution/MyFlow --output json
+```
+
+Removing an evaluator does NOT auto-clean `evaluatorRefs` arrays in eval sets that reference it. After removing, re-list eval sets and reconcile any stale refs:
+
+```bash
+uip maestro flow eval set list --path ./MySolution/MyFlow --output json
+```
+
+## Anti-patterns
+
+- **Don't pass `--type` in PascalCase.** `ExactMatch` fails; only `exact-match` is accepted.
+- **Don't omit `--model` on `llm-judge-*`.** The cloud worker fail-fasts before calling the LLM gateway; the resulting 500 is unhelpful.
+- **Don't hand-write evaluator refs unless you are repairing an eval set.** Omit `--evaluators` so the CLI links all current evaluators, or use `eval set add --evaluators <generated_id_or_file_base>`. Do not pass the evaluator display name.
+- **Don't copy evaluator JSON across projects without regenerating UUIDs.** `id` collisions silently corrupt resolution.
+- **Don't use deterministic evaluators (`exact-match`, `json-similarity`, `contains`) for natural-language outputs.** They will fail almost every test that paraphrases.
+- **Don't mix trajectory and output evaluators on the same data point unless the data point provides both `--expected` and `--expected-agent-behavior`.** Otherwise one of them scores against an empty placeholder.

package/skills/uipath-maestro-flow/references/evaluate/running-guide.md

@@ -0,0 +1,215 @@
+# Running Flow Evaluations
+
+`uip maestro flow eval run *` — start, monitor, inspect, and compare evaluation runs. All run commands require `uip login` and a Flow solution that already exists in Studio Web.
+
+> **Before running any of these:** read [upload-safety.md](upload-safety.md). The skill must NOT auto-run `uip solution upload` to satisfy the "solution must be in Studio Web" prerequisite. If the solution isn't in Studio Web, ask the user.
+
+## Start a Run
+
+```bash
+uip maestro flow eval run start \
+  --set "<set_name>" \
+  --path <flow_project> \
+  [--entry-point <entry>] \
+  [--solution-id <id>] \
+  [--project-id <id>] \
+  [--folder-key <key>] \
+  [--debug-mode <mode>] \
+  [--wait [--timeout <seconds>]] \
+  --output json
+```
+
+### Resolution order for `--solution-id` / `--project-id`
+
+The CLI auto-resolves these from project metadata in the working tree (typically `SolutionStorage.json` and the parent `.uipx`). Pass `--solution-id` or `--project-id` explicitly only when the working tree does not have those IDs (e.g., a freshly scaffolded local project that has never been uploaded).
+
+If the auto-resolution fails AND you have not passed explicit IDs, the start command will error. Do NOT respond by running `uip solution upload` automatically — see [upload-safety.md](upload-safety.md) for the right action.
+
+### `--folder-key`
+
+The Orchestrator folder key that scopes the run. Defaults to the user's personal workspace. Pass an explicit folder key when the eval should run in a shared/team folder.
+
+### `--wait` and `--timeout`
+
+Without `--wait`, the command returns immediately with `EvalSetRunId`. With `--wait`, the CLI blocks until the run reaches a terminal state (`Completed` or `Failed`) or `--timeout` elapses (default 600s, hardcoded by the CLI).
+
+`--timeout` only stops the local CLI from blocking. The server-side run continues regardless. Query progress with:
+
+```bash
+uip maestro flow eval run status <eval_set_run_id> \
+  --set "<set_name>" --path <flow_project> --output json
+```
+
+Polling cadence is not part of the public CLI contract — do not depend on a specific interval.
+
+### Output (no `--wait`)
+
+```json
+{
+  "Code": "FlowEvalRunStarted",
+  "Data": {
+    "EvalSetRunId": "a1b2c3d4-...",
+    "EvalSetName": "Smoke Tests",
+    "DataPoints": 3,
+    "Evaluators": 2
+  }
+}
+```
+
+### Output (`--wait`, on completion)
+
+The CLI emits a summary plus per-data-point results. Use `run results` for the same data on a completed run.
+
+## Check Status
+
+```bash
+uip maestro flow eval run status <eval_set_run_id> \
+  --set "<set_name>" \
+  --path <flow_project> \
+  --output json
+```
+
+```json
+{
+  "Code": "FlowEvalRunStatus",
+  "Data": {
+    "EvalSetRunId": "a1b2c3d4-...",
+    "Status": "Completed",
+    "Score": 0.86,
+    "Duration": "42.5s"
+  }
+}
+```
+
+Status values: `Pending`, `Running`, `Completed`, `Failed`. `Completed` and `Failed` are terminal.
+
+## Detailed Results
+
+```bash
+uip maestro flow eval run results <eval_set_run_id> \
+  --set "<set_name>" \
+  --path <flow_project> \
+  [--only-failed] \
+  [--verbose] \
+  [--export-format json|csv] \
+  --output json
+```
+
+Per-data-point fields: `DataPoint`, `Status`, `EvaluatorScores`, `Duration`, `Error` (plus `Justifications` when `--verbose`).
+
+### `--only-failed`
+
+Filter to data points that errored or scored a hard failure. Triage faster on large sets.
+
+### `--verbose`
+
+Include the LLM judge's free-text justification per evaluator. Essential when a score is surprising.
+
+### `--export-format <json|csv>`
+
+Write results to a file alongside the project (e.g., `eval-results-<timestamp>.json` or `.csv`). Useful for archiving or feeding into a dashboard.
+
+### Filtering with `--output-filter`
+
+`--output-filter` takes a JMESPath expression and applies it to the JSON payload before printing. Useful for triage:
+
+```bash
+# Show only data points named "checkout-flow"
+uip maestro flow eval run results <run_id> \
+  --set "Smoke Tests" --path ./MySolution/MyFlow --output json \
+  --output-filter '[?DataPoint==`checkout-flow`]'
+
+# Show only score and name per row
+uip maestro flow eval run results <run_id> \
+  --set "Smoke Tests" --path ./MySolution/MyFlow --output json \
+  --output-filter '[*].{name: DataPoint, score: EvaluatorScores}'
+```
+
+## List Past Runs
+
+```bash
+uip maestro flow eval run list \
+  --set "<set_name>" \
+  --path <flow_project> \
+  --output json
+```
+
+Per-row: `EvalSetRunId`, `Status`, `Score`, `DataPoints`, `Duration`, `CreatedAt`.
+
+## Compare Two Runs
+
+```bash
+uip maestro flow eval run compare <run_id_a> \
+  --compare-to <run_id_b> \
+  --set "<set_name>" \
+  --path <flow_project> \
+  --output json
+```
+
+Output:
+
+```json
+{
+  "Code": "FlowEvalRunComparison",
+  "Data": {
+    "RunA": { "Id": "...", "Score": 0.86, "Status": "Completed" },
+    "RunB": { "Id": "...", "Score": 0.80, "Status": "Completed" },
+    "ScoreDelta": 0.06,
+    "DataPoints": [
+      {
+        "DataPoint": "hello-test",
+        "ScoreA": 1.0, "ScoreB": 0.9, "Delta": "+0.1",
+        "StatusA": "Completed", "StatusB": "Completed"
+      }
+    ]
+  }
+}
+```
+
+`compare` aligns by data point `name` within the eval set. Comparing runs from different eval sets is not supported — the deltas would be meaningless.
+
+Use `compare` after each prompt or flow change to verify the change improved scores without regressing other data points.
+
+## Workflow Example
+
+```bash
+# 1. Verify the solution is in Studio Web (do NOT auto-upload — see upload-safety.md)
+#    If unsure, list runs first; absence of any error here implies the solution exists.
+uip maestro flow eval run list --set "Smoke Tests" --path ./MySolution/MyFlow --output json
+
+# 2. Start the run, block on completion
+uip maestro flow eval run start \
+  --set "Smoke Tests" \
+  --path ./MySolution/MyFlow \
+  --wait --timeout 600 --output json
+
+# 3. Inspect failures with justifications
+uip maestro flow eval run results <run_id> \
+  --set "Smoke Tests" \
+  --only-failed --verbose \
+  --path ./MySolution/MyFlow --output json
+
+# 4. After fixing the flow, re-run and compare
+uip maestro flow eval run start --set "Smoke Tests" --path ./MySolution/MyFlow --wait --output json
+uip maestro flow eval run compare <new_run_id> --compare-to <old_run_id> \
+  --set "Smoke Tests" --path ./MySolution/MyFlow --output json
+```
+
+## Failure Detection
+
+A data point is considered failed when any of these are true:
+
+- `Status` is `Failed`
+- `Error` is non-null
+- Any evaluator score type is `error`
+- An exact-match evaluator returned `false`
+
+Use `--only-failed` to filter to these rows. Use `--verbose` to read the justifications.
+
+## Anti-patterns
+
+- **Don't auto-run `uip solution upload`** when `eval run start` errors with a missing-solution error. Stop and ask the user — see [upload-safety.md](upload-safety.md).
+- **Don't depend on `--wait`'s polling cadence.** Treat as a black-box block.
+- **Don't compare runs from different eval sets.** `compare` aligns by data point name; cross-set deltas are meaningless.
+- **Don't rely on aggregate `Score` alone.** Inspect per-evaluator scores. A 0.86 aggregate can mask a high-similarity-but-wrong-trajectory failure.
+- **Don't keep retrying `eval run status` while `--wait` is still blocking from another shell.** Pick one — either `--wait` or polling status — to avoid race conditions on the same run ID.