@uipath/skills 1.201.0-preview.621 → 1.202.0-preview.635

package/skills/uipath-maestro-flow/references/evaluate/CAPABILITY.md

@@ -12,6 +12,7 @@ Capability index for `uip maestro flow eval` — evaluator CRUD (7 types), eval
 - Create evaluators (`exact-match`, `json-similarity`, `contains`, `llm-judge-*` types) for a Flow project
 - Create or remove eval sets, link them to evaluators, pin entry points
 - Add, list, or remove simulations on data points (`uip maestro flow eval simulation`)
+- Add, list, or remove child tool simulations on agent nodes (`--parent` flag)
 - Start an eval run on Studio Web, poll its status, fetch detailed results
 - Compare two eval runs to verify a change improved scores without regressions
 
@@ -20,7 +21,7 @@ For agent (`agent.json`) evaluations read the `uipath-agents` skill. For BPMN ev
 ## Critical rules
 
 1. **Check Flow eval CLI availability once.** Run `uip maestro flow eval --help --output json` before using eval commands. If it returns `unknown command 'eval'`, the installed CLI does not expose Flow eval yet. Stop, report that the user needs a CLI/tool version with Flow eval support, and do not spend turns searching npm packages or source bundles.
-2. **Never run `uip solution upload` automatically as part of an eval workflow.** The eval run requires the Flow solution to already exist in Studio Web, but uploading from the local working tree clobbers whatever is on Studio Web. If the project was pulled from Studio Web (`uip solution download`), edited locally in VS Code, or scaffolded on disk and never uploaded, an unprompted upload will overwrite or push unintended state. Ask the user explicitly before any `uip solution upload` — see [upload-safety.md](references/upload-safety.md).
+2. **Never run `uip solution upload` automatically as part of an eval workflow.** The eval run requires the Flow solution to already exist in Studio Web, but uploading from the local working tree clobbers whatever is on Studio Web. If the project was pulled from Studio Web (`uip solution download`), edited locally in VS Code, or scaffolded on disk and never uploaded, an unprompted upload will overwrite or push unintended state. Ask the user explicitly before any `uip solution upload` — see [upload-safety.md](upload-safety.md).
 3. **`--path` accepts a Flow project directory OR a solution directory containing exactly one Flow project.** If the solution holds multiple Flow projects, point `--path` at the specific project directory.
 4. **Local CRUD does not require login.** `add`, `remove`, `list` (data points / eval sets / evaluators) edit JSON on disk. Only `uip maestro flow eval run *` requires `uip login` and an existing Studio Web solution.
 5. **Pin a model on every LLM-judge evaluator.** Empty/missing `model` produces a cryptic 500 from the LLM gateway after retries. Pass `--model <name>` on `evaluator add` or set `model` in the JSON.
@@ -73,33 +74,34 @@ uip maestro flow eval run results <eval_set_run_id> \
 
 | Journey | Read |
 | --- | --- |
-| Look up any `uip maestro flow eval` subcommand syntax, flags, defaults, output codes | [commands-reference.md](references/commands-reference.md) |
-| Choose among the 7 evaluator types, write custom prompts, hand-write evaluator JSON | [evaluators-guide.md](references/evaluators-guide.md) |
-| Create eval sets, add data points, map `--inputs`/`--expected`/`--criteria` to evaluator types, attach files | [eval-sets-guide.md](references/eval-sets-guide.md) |
-| Start a Studio Web run, poll status, read results, export CSV/JSON, compare two runs | [running-guide.md](references/running-guide.md) |
-| Decide whether to call `uip solution upload` (almost always: don't auto-run; ask first) | [upload-safety.md](references/upload-safety.md) |
+| Look up any `uip maestro flow eval` subcommand syntax, flags, defaults, output codes | [commands-reference.md](commands-reference.md) |
+| Choose among the 7 evaluator types, write custom prompts, hand-write evaluator JSON | [evaluators-guide.md](evaluators-guide.md) |
+| Create eval sets, add data points, map `--inputs`/`--expected`/`--criteria` to evaluator types, attach files | [eval-sets-guide.md](eval-sets-guide.md) |
+| Start a Studio Web run, poll status, read results, export CSV/JSON, compare two runs | [running-guide.md](running-guide.md) |
+| Decide whether to call `uip solution upload` (almost always: don't auto-run; ask first) | [upload-safety.md](upload-safety.md) |
 
 ## Common tasks
 
 | I need to... | Read these |
 | --- | --- |
-| **Add an evaluator** | [evaluators-guide.md](references/evaluators-guide.md) + [commands-reference.md — Evaluators](references/commands-reference.md#evaluators) |
-| **Pick the right evaluator type** | [evaluators-guide.md — When to Pick Each Type](references/evaluators-guide.md#when-to-pick-each-type) |
-| **Create an eval set and pin an entry point** | [eval-sets-guide.md — Eval Set Lifecycle](references/eval-sets-guide.md#eval-set-lifecycle) |
-| **Add a data point with file attachments** | [eval-sets-guide.md — `--input-file`](references/eval-sets-guide.md#--input-file-keypath) |
-| **Set per-data-point criteria for trajectory evaluators** | [eval-sets-guide.md — `--criteria`](references/eval-sets-guide.md#--criteria) |
-| **Add a simulation to a data point** | [eval-sets-guide.md — Simulations](references/eval-sets-guide.md#simulations-on-data-points) + [commands-reference.md — Simulations](references/commands-reference.md#simulations) |
-| **Start a Studio Web eval run** | [running-guide.md — Start a Run](references/running-guide.md#start-a-run) |
-| **Poll run status without `--wait`** | [running-guide.md — Check Status](references/running-guide.md#check-status) |
-| **Inspect only failed data points** | [running-guide.md — Detailed Results](references/running-guide.md#detailed-results) (`--only-failed --verbose`) |
-| **Compare two runs side-by-side** | [running-guide.md — Compare Two Runs](references/running-guide.md#compare-two-runs) |
-| **Decide whether to call `uip solution upload`** | [upload-safety.md](references/upload-safety.md) |
-| **Look up the `eval` subcommand tree, flags, defaults, output codes** | [commands-reference.md](references/commands-reference.md) |
-| **My eval run failed** | [running-guide.md — Failure Detection](references/running-guide.md#failure-detection); for flow-level faults [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md) |
+| **Add an evaluator** | [evaluators-guide.md](evaluators-guide.md) + [commands-reference.md — Evaluators](commands-reference.md#evaluators) |
+| **Pick the right evaluator type** | [evaluators-guide.md — When to Pick Each Type](evaluators-guide.md#when-to-pick-each-type) |
+| **Create an eval set and pin an entry point** | [eval-sets-guide.md — Eval Set Lifecycle](eval-sets-guide.md#eval-set-lifecycle) |
+| **Add a data point with file attachments** | [eval-sets-guide.md — `--input-file`](eval-sets-guide.md#--input-file-keypath) |
+| **Set per-data-point criteria for trajectory evaluators** | [eval-sets-guide.md — `--criteria`](eval-sets-guide.md#--criteria) |
+| **Add a simulation to a data point** | [eval-sets-guide.md — Simulations](eval-sets-guide.md#simulations-on-data-points) + [commands-reference.md — Simulations](commands-reference.md#simulations) |
+| **Add a child tool simulation to an agent node** | [eval-sets-guide.md — Child Simulations](eval-sets-guide.md#child-simulations-agent-tool-simulation) + [commands-reference.md — Simulations](commands-reference.md#simulations) |
+| **Start a Studio Web eval run** | [running-guide.md — Start a Run](running-guide.md#start-a-run) |
+| **Poll run status without `--wait`** | [running-guide.md — Check Status](running-guide.md#check-status) |
+| **Inspect only failed data points** | [running-guide.md — Detailed Results](running-guide.md#detailed-results) (`--only-failed --verbose`) |
+| **Compare two runs side-by-side** | [running-guide.md — Compare Two Runs](running-guide.md#compare-two-runs) |
+| **Decide whether to call `uip solution upload`** | [upload-safety.md](upload-safety.md) |
+| **Look up the `eval` subcommand tree, flags, defaults, output codes** | [commands-reference.md](commands-reference.md) |
+| **My eval run failed** | [running-guide.md — Failure Detection](running-guide.md#failure-detection); for flow-level faults [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md) |
 
 ## Anti-patterns
 
-- **Don't auto-run `uip solution upload`.** Even when an eval run errors with "solution not found in Studio Web", stop and ask the user — see [upload-safety.md](references/upload-safety.md). The local project may be ahead of, or diverged from, Studio Web.
+- **Don't auto-run `uip solution upload`.** Even when an eval run errors with "solution not found in Studio Web", stop and ask the user — see [upload-safety.md](upload-safety.md). The local project may be ahead of, or diverged from, Studio Web.
 - **Don't hand-write `evaluatorRefs` unless you are repairing an eval set.** Prefer the default all-evaluators behavior so the CLI writes generated file refs, or pass generated evaluator ids/file refs explicitly. Do not pass evaluator display names to `--evaluators`.
 - **Don't pass `--type` in PascalCase.** Only kebab-case is accepted: `exact-match`, `json-similarity`, `contains`, `llm-judge-output`, `llm-judge-strict-json`, `llm-judge-trajectory`, `llm-judge-trajectory-simulation`.
 - **Don't depend on a specific `--wait` polling cadence.** Treat `--wait` as a black-box block; if you need precise progress, omit it and call `eval run status` yourself.
@@ -120,11 +122,11 @@ After a run completes, report:
 
 ### Evaluate-scoped
 
-- [commands-reference.md](references/commands-reference.md) — every `uip maestro flow eval` subcommand, flags, defaults, output `Code` enum
-- [evaluators-guide.md](references/evaluators-guide.md) — 7 evaluator types mapped to internal `uipath-*` IDs, JSON shapes, template variables
-- [eval-sets-guide.md](references/eval-sets-guide.md) — eval set + data point CRUD, `--inputs`/`--expected`/`--criteria`/`--input-file`/`--search-text`, simulations
-- [running-guide.md](references/running-guide.md) — run start/status/results/list/compare, JMESPath `--output-filter`, failure detection
-- [upload-safety.md](references/upload-safety.md) — the `solution upload` rule
+- [commands-reference.md](commands-reference.md) — every `uip maestro flow eval` subcommand, flags, defaults, output `Code` enum
+- [evaluators-guide.md](evaluators-guide.md) — 7 evaluator types mapped to internal `uipath-*` IDs, JSON shapes, template variables
+- [eval-sets-guide.md](eval-sets-guide.md) — eval set + data point CRUD, `--inputs`/`--expected`/`--criteria`/`--input-file`/`--search-text`, simulations
+- [running-guide.md](running-guide.md) — run start/status/results/list/compare, JMESPath `--output-filter`, failure detection
+- [upload-safety.md](upload-safety.md) — the `solution upload` rule
 
 ### Cross-capability (shared)
 

package/skills/uipath-maestro-flow/references/evaluate/{references/commands-reference.md → commands-reference.md}

@@ -141,25 +141,30 @@ Add or replace a simulation on a data point. If a simulation for `<component-id>
 | `--set <name>` | Yes | Eval set name or ID |
 | `--data-point <id>` | Yes | Data point name or ID |
 | `--strategy <strategy>` | Yes | `Llm` or `Static` |
-| `--component-type <type>` | Yes | Component type (e.g. `connector`, `agent`, `subflow`) |
+| `--component-type <type>` | No | Component type (e.g. `connector`, `agent`, `subflow`). Required unless `--parent` is used (defaults to `Node`). |
 | `--component-description <text>` | No | Human-readable label for the component |
 | `--simulation-instructions <text>` | No | LLM prompt describing what the component should return (for `Llm` strategy) |
 | `--mock-value <json>` | No | Static JSON output (for `Static` strategy) |
-| `--output-schema <json>` | No | JSON Schema describing the expected output shape; passed to the LLM to constrain its response. **Auto-resolved from the `.flow` file when omitted for `Llm` strategy** — fails if the node is not found or has no outputs. Pass explicitly to override. |
+| `--parent <component-id>` | No | Parent agent node component ID. When set, the simulation is added as a child tool simulation nested inside the parent agent node's simulation. If no parent simulation exists yet, one is auto-created (type `agent`, strategy `Llm`). `--component-type` defaults to `Node`. |
 | `--path <path>` | No | (see Common Options) |
 
 **Strategy guide:**
 
 | Strategy | When to use | Key flags |
 |----------|-------------|-----------|
-| `Llm` | Output should be realistic but non-deterministic | `--simulation-instructions`, `--output-schema` (auto-resolved) |
+| `Llm` | Output should be realistic but non-deterministic | `--simulation-instructions` (output schema auto-resolved) |
 | `Static` | Output is fixed and deterministic | `--mock-value` |
 
-**`--output-schema` auto-resolution for Llm simulations:** When omitted, the CLI reads the `.flow` file, finds the node by `<component-id>`, and derives the schema from the node's output definition (connector `outputJsonSchema`, agent `agentOutputVariables`, or `node.outputs`). Fails with an actionable error if the node is not found or has no outputs. Pass `--output-schema` explicitly to override. The JSON Schema is sent alongside `--simulation-instructions` to the LLM, telling it what shape the output must conform to. Without it the LLM generates free-form text. For a connector that returns `{ status, message }` you would pass:
+**Output schema auto-resolution:** The CLI always auto-resolves the output schema for both top-level and child (`--parent`) simulations:
 
-```bash
---output-schema '{"type":"object","properties":{"status":{"type":"string"},"message":{"type":"string"}}}'
-```
+- **Top-level simulations:** reads the `.flow` file, finds the node by `<component-id>`, and derives the schema from the node's output definition (connector `outputJsonSchema`, agent `agentOutputVariables`, or `node.outputs`).
+- **Child simulations (inline canvas agents):** finds the child tool node via edges in the `.flow` file and extracts its output schema.
+- **Child simulations (same-solution agents):** reads the inline agent's `agent.json` and matches the tool by name in the `resources[]` array.
+- **Child simulations (published agents):** calls the platform API (`simulatableComponents`) using the current login session to fetch the tool's schema. Requires `uip login`.
+
+Fails with an actionable error if the node/tool is not found or has no outputs.
+
+**Static mock value validation:** For `Static` child simulations, the CLI validates that `--mock-value` keys match the auto-resolved schema properties, catching shape mismatches early.
 
 Example — LLM strategy:
 
@@ -185,6 +190,33 @@ uip maestro flow eval simulation add agent-lookup \
   --path ./MySolution/MyFlow --output json
 ```
 
+Example — Child simulation (tool inside an agent node):
+
+```bash
+# Add a child tool simulation (Static). No separate parent simulation step
+# needed — --parent auto-creates the parent if it does not exist.
+# Output schema is auto-resolved from the agent's tool definitions.
+uip maestro flow eval simulation add Web_Search \
+  --parent agent-lookup \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --strategy Static \
+  --mock-value '{"results": [{"title": "Example", "url": "https://example.com"}]}' \
+  --path ./MySolution/MyFlow --output json
+
+# Add a child tool simulation (Llm). Output schema auto-resolved —
+# works for inline, same-solution, and published agents.
+uip maestro flow eval simulation add Send_Email \
+  --parent agent-lookup \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --strategy Llm \
+  --simulation-instructions "Return a success status with a generated messageId." \
+  --path ./MySolution/MyFlow --output json
+```
+
+The child's `<component-id>` is the tool's **runtime name** (e.g. `Web_Search`, `Send_Email`), not a `.flow` node ID. Child simulations are stored in the parent's `childSimulations` array.
+
 ### `uip maestro flow eval simulation list`
 
 List all simulations configured on a data point.
@@ -193,13 +225,22 @@ List all simulations configured on a data point.
 |------|----------|-------------|
 | `--set <name>` | Yes | Eval set name or ID |
 | `--data-point <id>` | Yes | Data point name or ID |
+| `--parent <component-id>` | No | Parent agent component ID. When set, lists child tool simulations on that parent instead of top-level simulations. |
 | `--path <path>` | No | (see Common Options) |
 
 ```bash
+# List top-level simulations
 uip maestro flow eval simulation list \
   --set "Smoke Tests" \
   --data-point "hello-test" \
   --path ./MySolution/MyFlow --output json
+
+# List child simulations on an agent node
+uip maestro flow eval simulation list \
+  --parent agent-lookup \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --path ./MySolution/MyFlow --output json
 ```
 
 ### `uip maestro flow eval simulation remove <component-id>`
@@ -210,13 +251,22 @@ Remove a simulation from a data point. Returns an error if no simulation with th
 |------|----------|-------------|
 | `--set <name>` | Yes | Eval set name or ID |
 | `--data-point <id>` | Yes | Data point name or ID |
+| `--parent <component-id>` | No | Parent agent component ID. When set, removes a child tool simulation from the parent instead of a top-level simulation. |
 | `--path <path>` | No | (see Common Options) |
 
 ```bash
+# Remove a top-level simulation
 uip maestro flow eval simulation remove connector-send-email \
   --set "Smoke Tests" \
   --data-point "hello-test" \
   --path ./MySolution/MyFlow --output json
+
+# Remove a child simulation from an agent node
+uip maestro flow eval simulation remove Web_Search \
+  --parent agent-lookup \
+  --set "Smoke Tests" \
+  --data-point "hello-test" \
+  --path ./MySolution/MyFlow --output json
 ```
 
 ## Run
@@ -296,6 +346,7 @@ The CLI emits a `Code` field on every JSON response. Useful when filtering or sc
 | `eval set add` / `list` / `remove` | `FlowEvalSetAdd` / `FlowEvalSetList` / `FlowEvalSetRemove` |
 | `eval evaluator add` / `list` / `remove` | `FlowEvalEvaluatorAdd` / `FlowEvalEvaluatorList` / `FlowEvalEvaluatorRemove` |
 | `eval simulation add` / `list` / `remove` | `FlowEvalSimulationAdd` / `FlowEvalSimulationList` / `FlowEvalSimulationRemove` |
+| `eval simulation add --parent` / `list --parent` / `remove --parent` | `FlowEvalChildSimulationAdd` / `FlowEvalChildSimulationList` / `FlowEvalChildSimulationRemove` |
 | `eval run start` (no `--wait`) | `FlowEvalRunStarted` |
 | `eval run start --wait` (summary) | `FlowEvalRunCompleted` |
 | `eval run status` | `FlowEvalRunStatus` |

package/skills/uipath-maestro-flow/references/evaluate/eval-sets-guide.md

@@ -0,0 +1,227 @@
+# Eval Sets and Data Points
+
+An eval set is a JSON file containing test cases (data points) and evaluators. A Flow project may have multiple sets, typically one per scenario.
+
+## Eval Set Lifecycle
+
+Run:
+
+```bash
+uip maestro flow eval set add "<set_name>" \
+  [--evaluators <ref1>,<ref2>] \
+  [--entry-point <node_id>] \
+  --path <flow_project> --output json
+
+uip maestro flow eval set list --path <flow_project> --output json
+
+uip maestro flow eval set remove "<id_or_name>" --path <flow_project> --output json
+```
+
+`--evaluators` accepts comma-separated evaluator IDs or generated file base names. If omitted, the set links to all evaluators present at creation and writes generated evaluator file refs such as `greeting-match-62c0793a.json`; prefer omission when using every current evaluator. Do not pass an evaluator display name such as `greeting-match`: this writes the display name into `evaluatorRefs`, which Studio Web eval runs cannot resolve. For an explicit list, run `uip maestro flow eval evaluator add/list --output json` and pass the generated ID/file ref. Evaluators added later are not auto-attached; rerun `set add` with explicit generated refs or carefully repair `evaluatorRefs` using CLI-created evaluator file refs.
+
+`--entry-point` is stored as `selectedEntrypoint` and accepts `/Main.bpmn#start` or a start node ID such as `start_a1b2c3d4`. `eval run start` uses it unless overridden with `--entry-point`. Pin it when multiple entry points exist; otherwise the first start node found is used.
+
+## Data Point Lifecycle
+
+Data points live inline in the eval set JSON. Run:
+
+```bash
+uip maestro flow eval add "<data_point_name>" \
+  --set "<set_name>" \
+  --inputs '{"...":"..."}' \
+  [--expected '{"...":"..."}'] \
+  [--criteria '{"<evaluator_id>": {...}}'] \
+  [--input-file <key>=<path>] \
+  [--search-text "<text>"] \
+  --path <flow_project> --output json
+
+uip maestro flow eval list --set "<set_name>" --path <flow_project> --output json
+
+uip maestro flow eval remove "<id_or_name>" --set "<set_name>" --path <flow_project> --output json
+```
+
+<a id="--criteria"></a>
+<a id="--input-file-keypath"></a>
+### Inputs, Expected Values, and Criteria
+
+`--inputs` must contain only keys matching the chosen entry point's declared input variables. If a variable is missing, add it or change the input JSON. To add a string input named `name`, run:
+
+```bash
+uip maestro flow variable add ./MySolution/MyFlow/MyFlow.flow name \
+  --direction in --type string --output json
+```
+
+Evaluator behavior:
+
+- `exact-match`: compares output verbatim, or `expected[targetKey]` when the evaluator has `--target-key`.
+- `json-similarity`: tree-compares with tolerance.
+- `contains`: does not use `--expected`; use `--search-text`.
+- `llm-judge-output`: substitutes `--expected` into `{{ExpectedOutput}}`.
+- `llm-judge-strict-json`: substitutes it per key into `{{ExpectedOutput}}`.
+- `llm-judge-trajectory`: uses it with `--expected-agent-behavior` set through `--criteria`.
+- `llm-judge-trajectory-simulation`: behaves like `trajectory`, with simulation context.
+
+`--criteria` contains per-evaluator overrides keyed by evaluator ID. Use it when evaluators in one set need different criteria:
+
+```json
+{
+  "<evaluator_ref_for_trajectory>": {
+    "expectedAgentBehavior": "Agent calls the weather tool with the user's location, then returns a one-sentence summary."
+  },
+  "<evaluator_ref_for_output>": {
+    "expectedOutput": {"summary": "Sunny in NYC, 72°F"}
+  }
+}
+```
+
+When omitted, output evaluators fall back to `--expected` and trajectory evaluators use defaults. Always provide `expectedAgentBehavior` for trajectory evaluators; scoring against an empty `{{ExpectedAgentBehavior}}` is meaningless.
+
+### File and Contains Inputs
+
+`--input-file <key>=<path>` is repeatable and attaches a staged file under the specified key for runtime use, including PDFs, CSVs, and images. Do not delete the source before the run completes.
+
+`--search-text` is for `contains` evaluators. It attaches the substring to test and is equivalent to writing `criteria` for that evaluator.
+
+## Eval Set JSON Shape
+
+```json
+{
+  "id": "<uuid>",
+  "name": "Smoke Tests",
+  "version": "1.0",
+  "evaluatorRefs": ["<evaluator-file-ref-1>.json", "<evaluator-file-ref-2>.json"],
+  "selectedEntrypoint": "/Main.bpmn#start",
+  "evaluations": [
+    {
+      "id": "<uuid>",
+      "name": "hello-test",
+      "inputs": {"message": "hello"},
+      "expectedOutput": {"reply": "Hello! How can I help you?"},
+      "evaluationCriterias": {
+        "<evaluator-ref-2>": {
+          "expectedAgentBehavior": "Agent responds with a friendly greeting."
+        }
+      }
+    }
+  ]
+}
+```
+
+Keep `version: "1.0"`; it identifies the new eval format. <!-- version-check-skip --> (eval-set file schema version, CLI-emitted by `eval set add` — not a `.flow` version)
+
+- `evaluatorRefs` references CLI-generated evaluator files; prefer CLI commands over hand-editing.
+- `selectedEntrypoint` pins the set's entry point; override it per run with `eval run start --entry-point`.
+- `evaluations[]` is the inline data-point list; order is informational.
+- `evaluationCriterias` (plural) maps per-data-point, per-evaluator overrides.
+
+## Aligning Inputs with the Flow Schema
+
+The data point's `inputs` must match the chosen entry point's input schema. Mismatches produce errors such as `Input "name" is not declared as an input variable in the flow`. Before adding data points, inspect `<flow>.flow` for `variables` entries with `direction: "in"`, or run:
+
+```bash
+uip maestro flow variable list <flow_file> --output json
+```
+
+## Simulations on Data Points
+
+Simulations replace selected nodes during an eval run with controlled outputs. Target a component by its `.flow` node `id`. Run:
+
+```bash
+uip maestro flow eval simulation add <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Llm \
+  --component-type connector \
+  --simulation-instructions "Pretend to send the email and return success." \
+  --path <flow_project> --output json
+
+uip maestro flow eval simulation add <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Static \
+  --component-type agent \
+  --mock-value '{"status":"ok"}' \
+  --path <flow_project> --output json
+
+uip maestro flow eval simulation list \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+
+uip maestro flow eval simulation remove <component-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+```
+
+| Strategy | Use | Required or relevant flags |
+|---|---|---|
+| `Llm` | Plausible, non-deterministic output | `--simulation-instructions` (output schema auto-resolved) |
+| `Static` | Identical output every run | `--mock-value <json>` |
+
+The output schema is always auto-resolved — for both top-level and child (`--parent`) simulations. Top-level reads the `.flow` node outputs; child simulations resolve from the `.flow` edges (inline agents), `agent.json` resources (same-solution agents), or the platform API (published agents, requires `uip login`).
+
+Simulations are stored inline in the data point's `simulations` array. Adding one for an existing `<component-id>` and data point replaces the existing simulation.
+
+### Child Simulations (Agent Tool Simulation)
+
+When a flow has an agent node, you can simulate individual tools inside that agent instead of mocking the whole agent as one unit. Use `--parent <agent-component-id>` to target a child tool. The child's `<component-id>` is the tool's runtime name (e.g. `Web_Search`, `Send_Email`), not a workflow node ID.
+
+```bash
+# Add a child tool simulation (Static — fixed output).
+# If no parent simulation exists for <agent-node-id>, one is auto-created
+# (type: agent, strategy: Llm).
+uip maestro flow eval simulation add Web_Search \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Static \
+  --mock-value '{"results": [{"title": "Example", "url": "https://example.com"}]}' \
+  --path <flow_project> --output json
+
+# Add a child tool simulation (Llm — prompt-guided).
+# Output schema is auto-resolved from the agent's tool definitions
+# (inline, same-solution, or published agents).
+uip maestro flow eval simulation add Send_Email \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --strategy Llm \
+  --simulation-instructions "Return a success status with a generated messageId." \
+  --path <flow_project> --output json
+
+# List child simulations on an agent node
+uip maestro flow eval simulation list \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+
+# Remove a child simulation
+uip maestro flow eval simulation remove Web_Search \
+  --parent <agent-node-id> \
+  --set "<set_name>" \
+  --data-point "<data_point_name>" \
+  --path <flow_project> --output json
+```
+
+No separate parent simulation step is needed — passing `--parent` auto-creates the parent simulation (type `agent`, strategy `Llm`) if it does not exist yet. This means a single command is enough to add a child tool simulation.
+
+When `--parent` is used, `--component-type` defaults to `Node` (the convention for child tool simulations). You can override it if needed.
+
+The output schema is auto-resolved for child simulations on all agent types:
+- **Inline canvas agents** (`uipath.agent.*`): resolved from the child tool node's outputs in the `.flow` file.
+- **Same-solution agents** (with `inputs.source`): resolved from the inline agent's `agent.json` resources.
+- **Published agents** (`uipath.core.agent.*`): resolved via the platform API (`simulatableComponents`). Requires `uip login`.
+
+For `Static` strategy, the CLI also validates that `--mock-value` keys match the resolved schema properties, catching shape mismatches before the eval run.
+
+Child simulations are stored in the parent's `childSimulations` array in the eval set JSON. Running `simulation add` with `--parent` twice for the same child `<component-id>` replaces the existing child simulation.
+
+## Anti-patterns
+
+- Do not hand-write data-point `id` UUIDs. Run `uip maestro flow eval add`; the CLI generates fresh UUIDs and maintains `evalSetId`.
+- Do not pass `--inputs` keys absent from the flow input schema; the CLI rejects them.
+- Do not set `--expected '{}'` while omitting `--criteria` for trajectory evaluators; both placeholders are empty and scoring is meaningless.
+- Do not delete attached input files before the run completes; the CLI references them until upload to Studio Web finishes.
+- Do not expect `--evaluators` on `set add` to update automatically; later evaluators are not retroactively linked.

package/skills/uipath-maestro-flow/references/evaluate/upload-safety.md

@@ -0,0 +1,94 @@
+# Upload Safety: Do Not Auto-Run `uip solution upload`
+
+Never run `uip solution upload` automatically during an evaluation workflow. Always ask the user first. This applies to locally created, Studio Web-downloaded, VS Code-authored, or otherwise divergent solutions, including when the CLI reports `solution-id could not be resolved` or a similar error.
+
+`uip solution upload` creates or overwrites the solution matched by the local `SolutionId` in Studio Web. Automatic upload can publish work in progress, overwrite concurrent Studio Web changes, or silently discard remote changes because local and remote state is not merged.
+
+## Read-Only Resolution Check
+
+This applies regardless of whether:
+
+- The local project was created via `uip maestro flow init` and never uploaded.
+- The local project was downloaded from Studio Web via `uip solution download` and edited locally.
+- The user is working in a VS Code-authored solution / personal workspace and the project may not match what is on Studio Web.
+- The CLI errors with `solution-id could not be resolved` or any variant.
+
+## Why
+
+`uip solution upload` is a write operation against Studio Web. It either creates a new solution or **overwrites** the existing one matched by the `SolutionId` in the local `.uipx` (bundling generates `SolutionStorage.json` from it; that file is read back only for a direct `.uis` upload) — the overwrite needs no flag and raises no prompt. The replaced contents are recorded as a restorable version (skipped entirely with `--no-snapshot`), but recovery is browser-only, after the fact, by the user — and it does not undo failure mode 1 below. None of this is a reason to skip asking. Three concrete failure modes if the skill auto-uploads:
+
+1. **The user is iterating locally** in VS Code or the filesystem and intended to test something **before** publishing. Auto-upload pushes work-in-progress to Studio Web where teammates and triggers may pick it up.
+2. **The user pulled the solution from Studio Web** to edit a small piece. Auto-upload sends the partial local state back, potentially **overwriting** changes another user made on Studio Web in the meantime.
+3. **The local solution diverged from Studio Web** (e.g., a teammate edited the solution on Studio Web while the dev was working locally). Auto-upload silently discards the remote-side changes — they are not merged.
+
+In all three cases the user loses work or surprises a teammate. The cost of pausing to ask is one prompt; the cost of an unwanted upload can be hours of recovery.
+
+## What To Do Instead
+
+When `eval run start` cannot resolve the solution:
+
+1. **Stop and ask the user.** Use plain language: "Your Flow solution doesn't appear to be in Studio Web (or its IDs aren't resolvable from the local working tree). I can't run an eval until it is. How do you want to proceed?"
+2. **Offer the options explicitly:**
+   - **Upload now** — the user runs (or asks the skill to run) `uip solution upload <SolutionDir> --output json`. They acknowledge that this will write to Studio Web.
+   - **Pass IDs explicitly** — the user provides `--solution-id` and `--project-id` for an existing Studio Web solution. The skill plumbs them through to `eval run start`.
+   - **Cancel** — they meant to test something else (e.g., `flow debug`).
+3. **Wait for an explicit decision.** Do not infer one from context, prior commands, or comments in the project.
+
+## How to Detect "Local Workspace or VS Code"
+
+There is no single CLI flag that says "this project is local-only." The signals to weigh, in priority order:
+
+1. **The `.uipx` carries an id Studio Web has never seen** (a fresh `solution init`, never uploaded). The upload would CREATE a new solution; the user might not want a new tenant entry.
+2. **`.vscode/` directory exists in the solution root.** Strong signal that the dev is authoring in VS Code; assume they are iterating locally.
+3. **The directory is under a workspace path the user has indicated they edit locally** (`~/Code/...`, `~/dev/...`, etc., or any path that is not the default Studio Web download location). Treat as local-first.
+4. **`uip solution upload` has never been recorded** in the recent shell history or in the conversation. If the skill cannot point to a prior explicit upload, do not assume the project is in Studio Web.
+
+If ANY of these signals is true, skip auto-upload entirely and ask.
+
+None of these signals separates a first import from an overwrite of a live solution: an id from a downloaded or copied project names a solution the upload would replace, and the local tree alone cannot tell the two apart. Treat both as writes to ask about.
+
+## How to Detect "Solution Already in Studio Web"
+
+The cheapest check is to attempt a read-only run command before doing anything else:
+
+```bash
+uip maestro flow eval run list --set "<set_name>" --path <flow_project> --output json
+```
+
+A successful response, including zero past runs, indicates that the solution is in Studio Web. A missing-solution or auth error means it is not uploaded for this workflow; ask before uploading. If `eval run start` fails with a resolution error, never auto-upload or retry by uploading.
+
+## If Evaluation Cannot Resolve the Solution
+
+1. Stop and ask: "Your Flow solution doesn't appear to be in Studio Web (or its IDs aren't resolvable from the local working tree). I can't run an eval until it is. How do you want to proceed?"
+2. Offer:
+   - **Upload now** — the user runs, or explicitly asks the skill to run, `uip solution upload <SolutionDir> --output json`; state that this writes to Studio Web.
+   - **Pass IDs explicitly** — the user provides `--solution-id` and `--project-id` for an existing Studio Web solution; pass them through to `eval run start`.
+   - **Cancel** — they intended to test something else, such as `flow debug`.
+3. Wait for an explicit decision. Never infer consent from context, prior commands, or project comments.
+
+## Local-Workspace Signals
+
+Treat the project as local-first and skip auto-upload if any signal is present:
+
+1. `SolutionStorage.json` is missing or has no `SolutionId`; upload would create a new solution.
+2. A `.vscode/` directory exists in the solution root.
+3. The directory is under a user-indicated local workspace path, such as `~/Code/...` or `~/dev/...`, rather than the default Studio Web download location.
+4. No prior `uip solution upload` appears in recent shell history or the conversation; without a recorded explicit upload, do not assume the project is in Studio Web.
+
+## Explicit Upload Consent
+
+When the user explicitly asks to upload, you may run:
+
+```bash
+uip solution upload <SolutionDir> --output json
+```
+
+Before running it, echo the exact command and warn that concurrent Studio Web edits may be overwritten. Afterward, report `SolutionId` and `DesignerUrl` from the output, surface the result, and only then proceed with the eval run. Do not silently chain the upload into the next step.
+
+## Anti-Patterns
+
+- Do not run `uip solution upload` to fix an eval-run error.
+- Do not treat "make the eval work" as upload consent.
+- Do not auto-retry an eval after solution resolution fails by uploading and rerunning.
+- Do not upload while another user may be editing the solution on Studio Web without warning about possible overwrites, even with consent.
+- Do not combine `flow debug` and `eval run` in one session against the same solution; each has its own Studio Web debug session, and mixing them can confuse run IDs and trigger unexpected uploads.

package/skills/uipath-maestro-flow/references/operate/CAPABILITY.md

@@ -27,27 +27,27 @@ Capability index for the lifecycle of a flow as a deployed asset. Operate owns e
 
 | Journey | Read |
 | --- | --- |
-| Publish a flow (Studio Web default, Orchestrator on request) | [ship.md](references/ship.md) |
-| Run a flow on demand or check progress | [run.md](references/run.md) |
-| Intervene in a running instance | [manage.md](references/manage.md) |
+| Publish a flow (Studio Web default, Orchestrator on request) | [ship.md](ship.md) |
+| Run a flow on demand or check progress | [run.md](run.md) |
+| Intervene in a running instance | [manage.md](manage.md) |
 
 ## Common tasks
 
 | I need to... | Read these |
 | --- | --- |
-| **Publish a flow to Studio Web** | [ship.md — Path 1](references/ship.md#path-1--studio-web-upload-default) |
-| **Deploy a flow to Orchestrator** (only if explicitly requested) | [ship.md — Path 2](references/ship.md#path-2--orchestrator-deploy-explicit-only) + [/uipath:uipath-solution](/uipath:uipath-solution) |
-| **Sync solution resource declarations** | [ship.md — Pre-flight](references/ship.md#pre-flight) (the `uip solution resources refresh` step) |
-| **Debug a flow end-to-end** | [run.md — Debug](references/run.md#debug--controlled-end-to-end-run) |
-| **Pass input arguments to `flow debug`** | [run.md — Debug](references/run.md#debug--controlled-end-to-end-run) (the `--inputs` flag) |
-| **Bind local files to file-typed inputs** | [run.md — Debug](references/run.md#debug--controlled-end-to-end-run) and [run.md — Process run](references/run.md#process-run--trigger-a-deployed-process) (same `--attachment <variableId>=<localPath>` flag on both, repeatable; `--attachment` overrides `--inputs` on key collisions) |
-| **Trigger a deployed process** | [run.md — Process run](references/run.md#process-run--trigger-a-deployed-process) |
-| **Check status of a running job** | [run.md — Job inspection](references/run.md#job-inspection--status-and-traces) |
-| **Stream verbose execution traces** | [run.md — Job inspection](references/run.md#job-inspection--status-and-traces) (use sparingly — see [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md)) |
-| **Pause a running instance** | [manage.md](references/manage.md) |
-| **Resume a paused instance** | [manage.md](references/manage.md) |
-| **Cancel an instance** | [manage.md](references/manage.md) |
-| **Retry a faulted instance** | [manage.md](references/manage.md) (after diagnosing root cause via [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md)) |
+| **Publish a flow to Studio Web** | [ship.md — Path 1](ship.md#path-1--studio-web-upload-default) |
+| **Deploy a flow to Orchestrator** (only if explicitly requested) | [ship.md — Path 2](ship.md#path-2--orchestrator-deploy-explicit-only) + [/uipath:uipath-solution](/uipath:uipath-solution) |
+| **Sync solution resource declarations** | [ship.md — Pre-flight](ship.md#pre-flight) (the `uip solution resources refresh` step) |
+| **Debug a flow end-to-end** | [run.md — Debug](run.md#debug--controlled-end-to-end-run) |
+| **Pass input arguments to `flow debug`** | [run.md — Debug](run.md#debug--controlled-end-to-end-run) (the `--inputs` flag) |
+| **Bind local files to file-typed inputs** | [run.md — Debug](run.md#debug--controlled-end-to-end-run) and [run.md — Process run](run.md#process-run--trigger-a-deployed-process) (same `--attachment <variableId>=<localPath>` flag on both, repeatable; `--attachment` overrides `--inputs` on key collisions) |
+| **Trigger a deployed process** | [run.md — Process run](run.md#process-run--trigger-a-deployed-process) |
+| **Check status of a running job** | [run.md — Job inspection](run.md#job-inspection--status-and-traces) |
+| **Stream verbose execution traces** | [run.md — Job inspection](run.md#job-inspection--status-and-traces) (use sparingly — see [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md)) |
+| **Pause a running instance** | [manage.md](manage.md) |
+| **Resume a paused instance** | [manage.md](manage.md) |
+| **Cancel an instance** | [manage.md](manage.md) |
+| **Retry a faulted instance** | [manage.md](manage.md) (after diagnosing root cause via [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md)) |
 | **Look up `solution` / `flow pack` / `flow debug` / `process` / `job` / `instance` CLI syntax** | [shared/cli-commands.md](../shared/cli-commands.md) |
 | **My flow run failed** | [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md) |
 
@@ -55,7 +55,7 @@ Capability index for the lifecycle of a flow as a deployed asset. Operate owns e
 
 - **Never run `solution upload` without `solution resources refresh` first.** Stale resource declarations cause runtime binding failures.
 - **Never default to Orchestrator deploy when the user said "publish".** "Publish" → Studio Web upload. Confirm explicitly before running `flow pack` + `solution publish`.
-- **Never run `flow debug` as a validation step.** Use `uip maestro flow validate` for correctness checking; debug is for end-to-end execution against real systems.
+- **Never run `flow debug` as a validation step, and never re-run a completed one to reshape its output.** Each run re-uploads the solution and executes the flow again against real systems; extract report fields from the payload the completed run already returned, and when that run faulted, read the cause from it first — see [diagnose/troubleshooting-guide.md — Step 0](../diagnose/troubleshooting-guide.md#step-0--read-the-cause-in-the-debug-output-you-already-have). Use `uip maestro flow validate` for correctness checking.
 - **Never `retry` a faulted instance without diagnosing the root cause first.** Triage via [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md) — read incidents, runtime variables, and the deployed asset. Then decide whether to retry, cancel, or re-author.
 - **Never start diagnosis from `job traces`.** Traces are last-resort verbose output. Begin with incidents — see [diagnose/CAPABILITY.md](../diagnose/CAPABILITY.md) for the priority ladder.
 
@@ -63,14 +63,14 @@ Capability index for the lifecycle of a flow as a deployed asset. Operate owns e
 
 ### Operate-scoped
 
-- [ship.md](references/ship.md) — Studio Web upload (default) and Orchestrator deploy (explicit)
-- [run.md](references/run.md) — debug, process run, job status/traces
-- [manage.md](references/manage.md) — instance lifecycle (pause, resume, cancel, retry)
+- [ship.md](ship.md) — Studio Web upload (default) and Orchestrator deploy (explicit)
+- [run.md](run.md) — debug, process run, job status/traces
+- [manage.md](manage.md) — instance lifecycle (pause, resume, cancel, retry)
 
 ### Cross-capability (shared)
 
 - [shared/cli-commands.md](../shared/cli-commands.md) — flat CLI lookup including `solution upload`, `solution resources refresh`, `flow pack`, `flow debug`, `flow process`, `flow job`, `flow instance`
-- [shared/cli-conventions.md](../shared/cli-conventions.md) — login states, FOLDER_KEY, UIPCLI_LOG_LEVEL, JSON output shape
+- [shared/cli-conventions.md](../shared/cli-conventions.md) — login states, FOLDER_KEY, UIP_LOG_LEVEL, JSON output shape
 - [shared/variables-and-expressions.md](../shared/variables-and-expressions.md) — `--inputs` JSON shape for `flow debug`
 
 For Orchestrator deployment via `uip solution publish`, see [/uipath:uipath-solution](/uipath:uipath-solution).