@uipath/skills 1.195.0 → 1.196.0

package/skills/uipath-review/references/agents/agents-coded-rules.md

@@ -0,0 +1,160 @@
+# Agents — Coded Judgment Rule Catalog
+
+Judgment rules for **coded** agents (Python — `main.py` + framework config). Each rule requires the agent to read source and reason — what a regex/AST-emulation/file-walk cannot decide reliably. Same row schema as elsewhere — see [`../rule-format.md`](../rule-format.md).
+
+> **This catalog is judgment-only.** Run `uip codedagent review "<PROJECT_DIR>" --output json` **first** (SKILL.md Step 2.5) — it returns the deterministic coded findings (pyproject/dependency/python-version gates, import & secret regex, framework symbol existence, bare-except, eval-run analysis, `.venv` packaging, git-tracked secrets) in the same rule format. Then apply the rules below, which the CLI cannot do.
+
+Read [`../rule-format.md`](../rule-format.md) and [`../rule-catalog-workflow.md`](../rule-catalog-workflow.md) first.
+
+Companion files:
+
+- [`agents-common-rules.md`](agents-common-rules.md) — judgment rule shared with low-code agents (schema-description informativeness)
+- [`agents-lowcode-rules.md`](agents-lowcode-rules.md) — low-code judgment rules (for the agent-builder coded layout that ships both `agent.json` and `main.py`)
+
+## Framework detection
+
+Many rules below gate on framework. Detect once and reuse:
+
+| Signal | Framework |
+|---|---|
+| `langgraph.json` at project root | `LANGGRAPH` |
+| `llama_index.json` | `LLAMAINDEX` |
+| `openai_agents.json` | `OPENAI_AGENTS` |
+| `google_adk.json` | `GOOGLE_ADK` |
+| `pydantic_ai.json` | `PYDANTIC_AI` |
+| `agent_framework.json` | `AGENT_FRAMEWORK` |
+| `uipath.json` with `.functions` and no framework config above | `FUNCTION` |
+
+Rules marked `(<FRAMEWORK> only)` in `trigger` skip on other frameworks.
+
+## Agent shapes
+
+- **Workflow** — one agent, one system prompt (zero for Simple Function), one tool surface, one entry point.
+- **Coded workflow** — multiple agents in one project; an orchestrator decides which agent handles which input. Detected when `create_react_agent(...)` is called ≥2 times, or there's a `StateGraph` supervisor over multiple agents, or OpenAI Agents `handoffs=[...]` over multiple `Agent` instances.
+
+Rules in the `## GeneralChecker` section tagged `(coded_workflow only)` skip on single-agent projects.
+
+## How to read this file
+
+One H2 section per checker class groups related rules for navigation. Every row's `detection_method` is the **judgment form**: read the named source, reason about it, emit when the criteria hold. Log the reasoning in the finding's `description`. Each section's last row is a `CODED_*_ISSUE` category bucket — use **only** when no specific rule fits the observation; do not bend specific rules to use a bucket.
+
+---
+
+## EvalsChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `CODED_EVAL_TRAJECTORY_SPECIFICITY` | warning | evals | Agent with 3+ tool calls has trajectory evaluators with generic `expectedAgentBehavior` | Read tools (AST-detected) + eval set. Assess: for agents with 3+ tool calls in a typical flow, do trajectory evaluators have `expectedAgentBehavior` specific enough to catch wrong tool sequences? `"Agent should process the input and return a response"` is too generic. Good: names tools, ordering, decisions. Emit per generic description. file = eval set JSON, element = evaluator id. | Rewrite `expectedAgentBehavior` to name the specific tool sequence and decisions. |
+| `CODED_EVAL_ARCHETYPE_FIT` | warning | evals | Chosen evaluators don't match the agent's archetype (LLM-only on tool agent, exact-match on free-form text, etc.) | Read agent source + eval evaluators. Classify the agent archetype: calculator/deterministic, text generator, multi-step orchestrator, API integration, classifier. Compare against [Archetype quick reference](#archetype-quick-reference) below. Emit per mismatch. file = eval set JSON. | Pick evaluators per the archetype reference: deterministic agents → exact-match/JSON-similarity; orchestrators → trajectory; classifiers → multiclass. |
+| `CODED_EVAL_SET_ORGANISATION` | info | evals | Mixed scenario concerns within one eval set, or unrelated agents share one set | Read eval sets. Assess: does one set mix happy-path + intentional-failure datapoints that distort the pass rate, or exercise unrelated agents in the same project? DO NOT fire on file-naming conventions (e.g. `set-1.json` is fine). Emit on content or ownership problems only. file = eval set JSON. | Split mixed sets by scenario type or agent; document the intent of each set in its filename or metadata. |
+| `CODED_EVAL_BEHAVIOR_DESCRIPTION` | warning | evals | `expectedAgentBehavior` is generic, not specific | Read each eval entry's `expectedAgentBehavior`. Assess: does it name the specific tools, steps, or decisions the agent should take? Strings like `"Agent should handle the request appropriately"` fail. file = eval set JSON, element = datapoint id. | Name the specific tools, steps, or decisions in the description. |
+| `CODED_EVAL_FORMAT_MISMATCH` | warning | evals | Expected-output string format doesn't match the agent's runtime serialization | Read eval `expectedOutput` strings + agent's output type (Pydantic / dataclass / JSON). Assess concrete mismatches: Python single-quoted dict (`"{'temperature': 25.0}"`) when runtime emits JSON; trailing-comma JSON; bytes-vs-string; regex anchors on a non-regex evaluator. Don't speculate — fire only on inspectable inconsistencies. file = eval set JSON, element = datapoint id. | Reformat `expectedOutput` to match the agent's actual serialization. |
+| `CODED_EVAL_COVERAGE_GAP` | warning | evals | Registered evaluator never referenced by an eval set, OR a code path never exercised by any datapoint | Read evaluator configs + eval-set `evaluationCriterias` keys. Assess: is each evaluator config referenced by ≥1 eval entry? Symmetrically, read agent source — does each declared code branch (platform-vs-local, error handler, rejection route) get exercised by ≥1 eval input? Emit per concrete gap. file = evaluator config or source file. | Either add eval entries that reference the evaluator / exercise the path, or remove the unused config. |
+| `CODED_EVAL_GROUND_TRUTH_MISLABELED` | warning | evals | Datapoint's expected class demonstrably contradicts input content | Read each datapoint's `.inputs` + `.expectedOutput` / `.expectedClass`. Assess: does the label unambiguously contradict input semantics (e.g., email body "I want to refund" labeled `expectedClass: "spam"`)? Borderline labels are eval-author judgment — skip. Emit on clear contradictions only. file = eval set JSON, element = datapoint id. | Correct the label, or split the datapoint into separate scenarios. |
+| `CODED_EVAL_DATASET_SINGLE_CLASS` | warning | evals | Multiclass evaluator paired with ≥3-datapoint dataset where all labels are identical | Read evaluator types + eval set labels. Assess: is the evaluator multi-class shape (`uipath-multiclass-classification`, custom balanced-accuracy with `classCounts: {…}`)? AND does the dataset have ≥3 datapoints all sharing the same label? Skip when evaluator is single-class by design (`uipath-binary-classification` for one-class detection) or when dataset has <3 datapoints. file = eval set JSON. | Add datapoints with the other classes, or switch to `uipath-binary-classification` for one-class detection. |
+| `CODED_EVAL_ISSUE` | warning | evals | Eval-infrastructure observation that no specific rule above fits | Use ONLY when no specific Evals rule fits. Walk all rules in this section first. Emit with concrete description + suggested fix. file = source of the observation. | (Defined per finding.) |
+
+### Archetype quick reference
+
+| Agent archetype | Primary evaluator | Secondary | Mismatch to flag |
+|---|---|---|---|
+| Calculator / deterministic | `uipath-exact-match` or `file://` | — | LLM judge on deterministic output with no deterministic evaluator |
+| Text generator / summariser | `uipath-llm-judge-output-semantic-similarity` | `uipath-contains` | `exact-match` on free-form text |
+| Multi-step orchestrator | `uipath-llm-judge-trajectory-similarity` | `uipath-tool-call-order` | No trajectory evaluator |
+| API integration | `uipath-json-similarity` or `file://` | `uipath-exact-match` | No structured comparison |
+| Classifier | `uipath-binary-classification` / `uipath-multiclass-classification` or `file://` | — | `exact-match` on classification output |
+
+LLM-judge-only is acceptable when the agent has no tools, no classification output, and emits free-form text. Otherwise prefer pairing with a deterministic baseline.
+
+---
+
+## SchemaChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `CODED_SCHEMA_COMPLETENESS` | warning | schema | `StateGraph` input/output schema is technically present but the fields are insufficient for the agent's contract | Read `StateGraph(input=..., output=...)` (or `_schema=` variants). Read the agent's actual behavior. Assess: are declared input/output fields meaningful and sufficient? An agent that needs `customer_history` + `recent_invoices` but only declares `query: str` is incomplete. file = source file. | Expand the schema to carry the context the agent actually needs / returns. |
+| `CODED_SCHEMA_FIELD_NO_VALIDATION` | warning | schema | Constrained-value field declared as bare `str` (no `Literal`/`Enum`/regex pattern) | Read Pydantic / dataclass models. Assess per field: does the name suggest a constrained set (`category`, `status`, `severity`, `role`, `intent`, `classification`, `priority`)? AND is the type bare `str` (no `Literal`, no `Enum`, no validator)? AND does the agent's logic visibly map to a small enumerated set? Emit per concrete case. file = source file, element = `<Model>.<field>`. | Type the field as `Literal["a", "b", "c"]` or `Enum`, or add `Field(..., pattern="...")`. |
+| `CODED_OUTPUT_ENUM_MISSING_ON_CLASSIFIER` | warning | schema | Classifier-shaped output field declared without enum | Read output schema (Pydantic field, `StateGraph` output annotation, dataclass). Identify classifier-shape fields by name (`class`, `classification`, `label`, `category`, `intent`, `severity`, `priority`, `status`) with `type: str`. Assess: is there a `Literal[...]` / `Enum` / pattern constraint? Both this rule and `CODED_SCHEMA_FIELD_NO_VALIDATION` can fire on the same field — the more-specific firing clarifies the issue. file = source file, element = field name. | Add `Literal[...]` / `Enum` / pattern constraint to the classifier output field. |
+| `CODED_SCHEMA_ISSUE` | warning | schema | Schema observation that no specific rule above fits | Use ONLY when no specific Schema rule fits. Emit with concrete description + suggested fix. file = source of the observation. | (Defined per finding.) |
+
+---
+
+## ToolsChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `CODED_TOOL_DOCSTRING_QUALITY` | warning | tools | Tool docstring missing, generic, or missing parameter mention | Read each tool-surface function (passed to `bind_tools`, `@tool`-decorated, registered via `Tool(...)`). Assess docstring: is it missing or <20 chars? Does it state what the tool does in plain language? Does it mention every declared parameter? Is it generic boilerplate? Fire when **at least one** of these holds. Don't fire when the docstring names every param and either describes the return shape or gives a usage cue. file = source file, element = function name. | Write a docstring that names every parameter, describes the return shape, and gives a usage cue (preconditions, side effects, example). |
+| `CODED_PROMPT_TOOL_COVERAGE` | warning | tools | Fewer than half the registered tools are mentioned in the system prompt | Read system prompt (passed to `UiPathChat`/`UiPathAzureChatOpenAI`/`Agent(instructions=...)`). Read tool registrations. Assess: is each tool mentioned by name or described by purpose? Emit when `<50%` are referenced. file = source file. | Reference each tool by name in the system prompt with guidance on when to use it. |
+| `CODED_PROMPT_REFERENCES_NONEXISTENT_TOOL` | warning | tools | System prompt names a tool that's not registered | Read system prompt. Extract tool names cited (backticked, quoted, "the X tool"). Build registered tool set (`@tool`-decorated, `Tool(...)` registered, `bind_tools` entries). Emit per cited name not in registered set. Skip prose references like "use a search tool" without specific identifiers. file = source file, element = missing tool name. | Either rename the tool, register the cited tool, or remove the prompt reference. |
+| `CODED_TOOLS_ISSUE` | warning | tools | Tools observation that no specific rule above fits | Use ONLY when no specific Tools rule fits. Emit with concrete description + suggested fix. file = source of the observation. | (Defined per finding.) |
+
+---
+
+## GuardrailsChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `CODED_GUARDRAILS_ISSUE` | warning | guardrails | Guardrails / safety / PII observation that no specific rule fits | Use ONLY when no specific rule fits. Coded agents have most safety concerns covered under `## SecurityChecker` (`CODED_PROMPT_USER_INPUT_UNSANITIZED`, `CODED_PII_IN_TRACES`) and the deterministic secret/`.env` checks in the review CLI. Reach here only for guardrail / policy observations none of those cover. | (Defined per finding.) |
+
+---
+
+## CodeChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `PYPROJECT_PLACEHOLDER` | warning | code | `name`, `description`, or `authors` are leftover scaffold defaults, not real metadata | Read `pyproject.toml` `[project]` `name` / `description` / `authors`. Assess: are these real project metadata or scaffold defaults left unchanged (`my-agent`, `app`, `Add your description here`, `Your Name <you@example.com>`)? Code cannot reliably tell a real value from a leftover placeholder — this is a semantic read. file = `pyproject.toml`, element = field name. | Replace placeholder values with project-specific metadata. |
+| `LANGGRAPH_GRAPH_NOT_COMPILED` | error | code | Exported graph symbol is a `StateGraph(...)` never `.compile()`d (`LANGGRAPH` only) | Read the module `langgraph.json` points to. Assess: is the exported graph symbol the result of a `.compile()` call? A top-level `graph = StateGraph(...)` with no trailing `.compile()` (on the same or a later line, directly or via a helper) ships an uncompiled graph the runtime can't execute. Reason about aliasing/indirection a regex would miss. file = source file, element = graph symbol. | Chain `.compile()` onto the `StateGraph(...)` (or compile it before export). |
+| `STATEGRAPH_MISSING_INPUT_OUTPUT` | warning | code | `StateGraph(...)` built without explicit input/output schemas (`LANGGRAPH` only) | Read the `StateGraph(...)` construction. Assess: are explicit `input=`/`input_schema=` and `output=`/`output_schema=` schemas passed? Without them the graph's external contract is the full internal state, leaking intermediate fields. file = source file. | Pass `input_schema=` and `output_schema=` to `StateGraph(...)`. |
+| `LLAMAINDEX_SYNC_STEP` | error | code | `@step`-decorated method is not `async def` (`LLAMAINDEX` only) | Read the workflow source. Assess: is every `@step`-decorated method declared `async def`? A synchronous `@step` breaks the LlamaIndex workflow runtime. file = source file, element = method name. | Add `async` to the step method. |
+| `OPENAI_AGENTS_UNSUPPORTED_FEATURE` | error | code | LangGraph-only feature called in an OpenAI Agents project (`OPENAI_AGENTS` only) | Read source. Assess: does it call `interrupt`, `MemorySaver`, or `InvokeProcess` — LangGraph-specific features unsupported under OpenAI Agents? Resolve import aliasing (`from x import interrupt as y`) the way a regex cannot. file = source file, element = unsupported call. | Remove the unsupported call — these are LangGraph-specific. |
+| `CODED_HELPER_TRACING` | info | code | LangGraph helpers outside the graph lack `@traced()` (`LANGGRAPH` only) | Read source. Identify functions called outside graph nodes (post-processing, formatting). Assess: do they have `@traced()`? Without it, helper execution won't appear in traces. Emit per untraced helper. file = source file, element = function name. | Decorate helper functions outside the graph with `@traced()`. |
+| `CODED_DEAD_CODE` | info | code | Unreachable conditional branch in agent logic | Read source. Assess: are there conditionals where one branch can never execute (`if x > y: do_a; if x > 0: do_b` — second branch unreachable when `x > y > 0`)? Skip merely-redundant-but-reachable defensive checks. Emit per concrete unreachable branch. file = source file, element = `<function>:<line>`. | Remove the unreachable branch, or fix the condition that makes it unreachable. |
+| `CODED_LLM_OUTPUT_UNVALIDATED` | warning | code | LLM completion used downstream without value-space validation | Read code paths that consume LLM responses (`response.content`, `chat_async(...).strip()`, `client.invoke(...)`). Assess per use site: (a) is the output constrained (`Literal`, `Enum`, fixed string set, JSON schema)? AND (b) is there no validation between the LLM call and the use site? Skip when consumed via `Output.model_validate(...)` or `Enum`-checked. Emit per concrete case. file = source file. | Validate the LLM response against the expected schema/Enum before downstream use. |
+| `CODED_ERROR_HANDLING` | warning | code | Risky external-call site without try/except, fallback, retry, or error surfacing | Read source. Identify external boundaries that can fail in production (LLM `ainvoke`, retriever `ainvoke`, attachment / queues / entities / processes API, HITL `interrupt`, HTTP/DB). Assess per call site: is there a `try/except`, fallback, retry, or error-state surfacing? Skip pure-Python helpers. The multi-agent supervisor variant is `CODED_MULTI_AGENT_ERROR_HANDLING`. Emit per concrete case. file = source file. | Wrap the call in try/except with a fallback path or surface the error in the agent's output state. |
+| `CODED_INVOKEPROCESS_NO_FALLBACK` | warning | code | `interrupt(InvokeProcess(...))` call without branching on `result.status` | Read source. Fire only when source actually imports or calls `InvokeProcess` / `InvokeProcessEvent`. Assess: does the code branch on `status` (`if status ==`, `match` on `.status`, `result.get('status')`)? Without it, `failed`/`faulted` returns malformed data. For general LLM / HTTP / SDK retry use `CODED_ERROR_HANDLING` instead. file = source file. | Branch on `result.status`: handle `"success"` / `"failed"` / `"faulted"` distinctly. |
+| `CODED_CODE_ISSUE` | warning | code | Code observation that no specific rule above fits | Use ONLY when no specific Code rule fits. Emit with concrete description + suggested fix. file = source of the observation. | (Defined per finding.) |
+
+---
+
+## GeneralChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `CODED_FRAMEWORK_FIT` | info | general | Framework chosen is a genuine mismatch for the task | Read source + framework config. Assess: is the framework appropriate? Real mismatches: LangGraph used for a deterministic pipeline with no branching; Simple Function used for a multi-step orchestrator that needs persistent state; LlamaIndex used for an agent with no retrieval. **Fire ONLY on real mismatch** — never as a commentary slot when the framework genuinely fits. file = framework config. | Switch to the framework that matches the task pattern. |
+| `CODED_UIPATH_JSON_FIELD_DRIFT` | warning | general | `uipath.json` field demonstrably wrong (`entryPoint` mismatch, `isConversational` wrong, `includeUvLock: false`, …) | Read `uipath.json` + source. Assess per field: does the declared value contradict the actual code shape or break the runtime contract? `entryPoint: main.py:classify` when the function is `main`; `isConversational: true` on a stateless agent; `packOptions.includeUvLock: false` (deployment-blocker). Skip stylistic preferences. file = `uipath.json`, element = field name. | Correct the field to match the code reality. |
+| `CODED_DOC_CODE_DRIFT` | info | general | README / docstring describes a feature the code doesn't implement | Read README + main docstrings. Assess: do they mention integrations, evaluators, or behaviors absent from source? Trust hazard for future contributors. Skip outdated inline comments inside functions (different concern) and "Future work" / "Roadmap" sections. file = README or docstring source. | Update the doc to match what the code actually does, or implement the documented feature. |
+| `CODED_PROMPT_QUALITY` | warning | general | System prompt has internal contradictions, circular logic, or ambiguous instructions | Read any system prompt embedded in coded source (`UiPathChat`/`UiPathAzureChatOpenAI`, `Agent(instructions=...)`, `ChatPromptTemplate` strings). Assess: (a) internal contradictions ("Always respond in JSON" + "Write as a friendly paragraph"); (b) circular logic (rule X says "see Y", Y says "see X"); (c) ambiguous instructions two readers would interpret differently. DO NOT fire merely because the prompt is long. Bar: literal reading produces inconsistent behaviour. file = source file. | Reconcile contradictions, break circular references, or disambiguate. |
+| `CODED_MULTI_AGENT_HUMANMESSAGE_NAME` | info | general | (coded_workflow only) Worker `HumanMessage(...)` returns lack `name="<worker_name>"` | Read worker functions in a multi-agent / supervisor setup. Assess: do they return `HumanMessage(content=..., name="<worker_name>")`? Without `name`, the supervisor cannot attribute responses. file = source file, element = worker name. | Add `name="<worker_name>"` to each worker's `HumanMessage(...)`. |
+| `CODED_MULTI_AGENT_ROUTING_COHERENCE` | warning | general | (coded_workflow only) Supervisor routing description doesn't match each worker's actual capability | Read supervisor routing prompt + each worker's system prompt. Assess: does the routing description match what each worker actually does? "Researcher handles fact-finding" + worker actually generates code → routing will be unreliable. file = supervisor source. | Reconcile routing description with worker capabilities; or rename / repurpose workers. |
+| `CODED_MULTI_AGENT_ERROR_HANDLING` | warning | general | (coded_workflow only) Multi-agent project lacks supervisor-level error handling | Read supervisor. Assess: project has ≥2 sub-agents (workers / handoffs) AND supervisor lacks fallback edges, try/except, or error states in routing logic. An unhandled worker exception terminates the entire graph. For single-agent error handling use `CODED_ERROR_HANDLING`. file = supervisor source. | Add fallback edges / try-except at the supervisor level. |
+| `CODED_MULTI_AGENT_CROSS_SCHEMA` | warning | general | (coded_workflow only) `sdk.processes.invoke(...)` targeting a sibling agent has cross-schema mismatch | Read source. For each `sdk.processes.invoke()` targeting another agent in the same repo, read both the call's `input_arguments` and the target agent's declared input schema. Assess: are keys + types consistent? Emit per concrete mismatch. file = source file. | Reconcile `input_arguments` with the target agent's input schema. |
+| `CODED_SUB_AGENT_PROMPT_DUPLICATION` | warning | general | (coded_workflow only) Two or more sub-agent prompts are substantively the same (paraphrase-equivalent) | Read each sub-agent's system prompt (`create_react_agent` workers, `Agent[Ctx]` instances). Assess pairwise: are any two prompts paraphrase-equivalent (not literal byte-match)? Routing becomes unreliable when worker capabilities are indistinguishable. file = source file, element = `<agent_a>,<agent_b>`. | Differentiate prompts so each worker has a distinct, scoped responsibility. |
+| `CODED_GENERAL_ISSUE` | warning | general | General-category observation that no specific rule above fits | Use ONLY when no other category fits — repository hygiene, documentation drift, project structure. Emit with concrete description + suggested fix. file = source of the observation. | (Defined per finding.) |
+
+---
+
+## SecurityChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `CODED_PROMPT_USER_INPUT_UNSANITIZED` | warning | security | User-controlled input interpolated into LLM prompt without sanitization | Read source. Identify user-controlled fields (function params from `input.X`, request body, attachment content, conversational history). Assess per use: (a) does the field reach a prompt template via concatenation / f-string (`f"Classify this email: {input.email_body}"`)? AND (b) is no `prompt_injection` guardrail registered? AND (c) is no sanitization helper (allowlist match, length cap, `html.escape`, regex-strip) between input and prompt? file = source file. | Sanitize the input before interpolation, or register a `prompt_injection` guardrail. |
+| `CODED_PII_IN_TRACES` | warning | security | `@traced()` on a function receiving PII-suggesting fields without redaction | Read `@traced`-decorated function signatures. Assess: do params have names suggesting PII (`email_body`, `customer_email`, `personal_*`, `*_ssn`, `customer_name`)? AND does the decorator lack `hide_input=True` / `input_processor=...`? Skip when the function only handles non-PII data (item IDs, timestamps). Emit per concrete case. file = source file, element = function name. | Pass `hide_input=True` or `input_processor=<redaction_fn>` to `@traced(...)`. |
+
+---
+
+## RuntimeQuirksChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `LIST_ACCUMULATOR_NOT_FORWARDED` | warning | runtime | LangGraph reducer-typed state key returned by a node without forwarding prior items (`LANGGRAPH` only) | Read the LangGraph state class + every node return. Assess: for each state key typed `Annotated[list[...], operator.add]` (a reducer), does any node return that key with only its new items instead of forwarding prior state? This is a whole-program dataflow read with no reliable regex — reason about it. **Advisory, not gating.** file = source file, element = state key. | Forward the accumulator: return `{"items": [*state.get("items", []), "new"]}` instead of `{"items": ["new"]}`. |
+
+---
+
+## What this catalog cannot do
+
+The agent applies these rules from the project source as it is *checked into the repo*. It cannot:
+
+- Verify runtime behavior (whether the LLM follows the prompt, picks the right tool, routes correctly).
+- Verify multi-agent correctness at runtime — routing logic can look structurally sound and still produce wrong results.
+- Verify external dependencies (process schemas, connection IDs, index names).
+- Catch deep code logic bugs (field-name mismatches, arithmetic errors) — those are for linters and human code review.

package/skills/uipath-review/references/agents/agents-common-rules.md

@@ -0,0 +1,32 @@
+# Agents — Common Rules (both formats)
+
+Judgment rules that apply to **both** coded and low-code agents. Read [`../rule-format.md`](../rule-format.md) and [`../rule-catalog-workflow.md`](../rule-catalog-workflow.md) first.
+
+> **This catalog is judgment-only.** Deterministic agent checks (eval counts, tool counts, schema-property presence, set-membership, regex, run-artifact analysis) run in the `uip agent review` / `uip codedagent review` CLI, which the reviewer invokes **first** (SKILL.md Step 2.5). The CLI returns those findings in the same rule format. This file holds only rules that require an LLM to read source and reason — what a regex or count cannot decide reliably.
+
+Companion files:
+
+- [`agents-lowcode-rules.md`](agents-lowcode-rules.md) — low-code judgment rules
+- [`agents-coded-rules.md`](agents-coded-rules.md) — coded judgment rules
+
+---
+
+## SchemaChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `SCHEMA_NO_DESCRIPTIONS` | judgment | schema | Input/output schema properties lack **informative** descriptions | Locate the schema (see [Schema location](#schema-location)). Read each property's `description`. Assess — not mere presence: is each description informative enough for the LLM to populate / consume the field correctly and for Studio Web to render a useful hint? A field named `id` described as `"the id"` adds no signal; a missing description on a non-obvious field is worse. Emit when a meaningful share of properties are undescribed or described with content that restates the field name. file = schema source. (The deterministic `>50%-missing` count runs in the review CLI; this rule judges informativeness.) | Add a `description` to each property that states what it holds, valid values, and format — Studio Web shows these as field hints and the LLM uses them as semantic signals. |
+
+### Schema location
+
+- **Coded** — Read `entry-points.json` if present; else `uipath.json` → `.entryPoints[0].input` / `.entryPoints[0].output`. The finding's `file` points to whichever source was read.
+- **Low-code normalized** — Read the normalized JSON. Use `.input_schema` and `.output_schema`.
+- **Low-code agent-builder** — Read `entry-points.json`. Use `.entryPoints[0].input` and `.entryPoints[0].output`.
+
+Assess input and output schema independently — emit a separate finding per schema when each is weak.
+
+---
+
+## What this catalog cannot do
+
+The agent applies these rules from the project source as it is *checked into the repo*. It cannot verify runtime behaviour (whether the LLM follows the prompt, whether tool selection is correct, whether guardrails fire) or validate live IDs (connection IDs, index names, recipients) — those require running the agent or calling APIs.

package/skills/uipath-review/references/agents/agents-lowcode-rules.md

@@ -0,0 +1,128 @@
+# Agents — Low-code Judgment Rule Catalog
+
+Judgment rules for **low-code** agents (`agent.json`). Each rule requires the agent to read source and reason — what a regex/count/schema-walk cannot decide reliably. Same row schema as elsewhere — see [`../rule-format.md`](../rule-format.md).
+
+> **This catalog is judgment-only.** Run `uip agent review "<PROJECT_DIR>" --output json` **first** (SKILL.md Step 2.5) — it returns the deterministic low-code findings (structural gates, schema-property presence, placeholder cross-refs, eval counts/diversity, version/runtime/location checks) in the same rule format. Then apply the rules below, which the CLI cannot do.
+
+Read [`../rule-format.md`](../rule-format.md) and [`../rule-catalog-workflow.md`](../rule-catalog-workflow.md) first.
+
+Companion file:
+
+- [`agents-common-rules.md`](agents-common-rules.md) — judgment rule shared with coded agents (schema-description informativeness)
+
+## Layouts
+
+Low-code agents exist in two layouts. Detect first, then apply the right `Read` path in each rule.
+
+| Layout | Identifier | Key files |
+|---|---|---|
+| **Normalized** | A single JSON file with snake_case top-level keys (`system_prompt`, `tools`, `datasets`, `input_schema`, `output_schema`, `user_prompt`) | The single JSON file |
+| **Agent-builder** | `agent.json` at project root with camelCase keys + sibling `entry-points.json`, `project.uiproj`, `resources/<Name>/resource.json` | `agent.json`, `entry-points.json`, `resources/*/resource.json`, `evals/eval-sets/*.json`, `evals/evaluators/*.json` |
+
+Rules tagged `(agent-builder only)` skip silently on the normalized layout; `(normalized only)` skip on agent-builder. Untagged rules apply to both with layout-aware `detection_method`.
+
+## How to read this file
+
+One H2 section per checker class (`EvalsChecker`, `SchemaChecker`, `ToolsChecker`, `GuardrailsChecker`, `GeneralChecker`) groups related rules for navigation. Every row's `detection_method` is the **judgment form**: read the named source, reason about it, emit when the criteria hold. Log the reasoning in the finding's `description`.
+
+---
+
+## EvalsChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `LC_EVAL_GROUND_TRUTH_SWAPPED` | warning | evals | ≥2 eval datapoints share substantially overlapping input shape but disagree on `expectedOutput` (suggests label swap) | Read `evals/eval-sets/*.json`. For each pair of datapoints, compare `inputs` content overlap (e.g., `email_subject` + `email_body` substring overlap > 80%). Assess: when inputs are near-identical but expected outputs differ, this is a likely label swap (the *opposite* failure pattern from the deterministic duplicate-output check). Emit one finding per swapped pair, element = `<id_1>,<id_2>`. file = eval set JSON. | Inspect both datapoints; correct the swapped label, or split them into clearly distinct scenarios. |
+| `LC_EVAL_JUDGE_FOR_CLOSED_CLASS` | warning | evals | LLM-judge evaluator applied to a closed-class output field | Read eval set's `evaluatorRefs`. Read agent's `outputSchema.properties`. Assess: if any output property has `.enum` (closed set) AND an evaluator referenced by the eval set is an LLM-judge (`evaluatorTypeId: "uipath-llm-judge-output-semantic-similarity"` or similar), the LLM judge is the wrong tool — `uipath-multiclass-classification` / `uipath-exact-match` gives 100% of the signal at zero variance. Emit one finding per (eval-set, judge-evaluator) pair. file = eval set JSON. | Replace the LLM-judge with `uipath-multiclass-classification` or `uipath-exact-match` for closed-class outputs. |
+
+---
+
+## SchemaChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `LC_OUTPUT_FORMAT_PROMPT_DRIFT` | warning | schema | System prompt instructs an output shape that conflicts with `outputSchema` | Read system prompt + `outputSchema`. Assess: does the prompt instruct a shape (`"respond with a single sentence"`, `"answer in markdown"`) that conflicts with the schema's structure (structured JSON, flat string, etc.)? Emit when prompt-vs-schema drift is present. file = system prompt source. | Align the prompt's output-format instructions with `outputSchema` — pick one source of truth and have the other reference it. |
+| `LC_INPUT_SCHEMA_OVERLAP` | warning | schema | Input schema has ≥2 fields with semantically overlapping names or descriptions | Read `inputSchema.properties`. Assess: are there fields with semantically overlapping names (`query` vs `question` vs `text` vs `prompt` vs `input`) or descriptions? Emit when ≥2 fields could reasonably hold the same content and the LLM would have to disambiguate at runtime. file = schema source, element = `<field_a>,<field_b>`. | Merge overlapping fields, or differentiate them with clear names + descriptions that scope each one. |
+| `LC_OUTPUT_ENUM_VS_PROMPT_DRIFT` | warning | schema | Output enum disagrees with the prompt's category list (either side has extras) | Read `outputSchema.properties.<X>.enum` and the system prompt's classification list (`"Classify into: A, B, C"`). Assess: do the two sets agree? Emit when either has extras: prompt names a class the schema doesn't accept (runtime validation rejects), OR schema accepts a class the prompt never mentions (LLM never produces it). file = schema or prompt source. | Reconcile the prompt's class list with `outputSchema.enum`. |
+| `LC_OUTPUT_CLASSIFIER_NO_ENUM` | warning | schema | Classifier-shaped agent has no `enum` constraint on the primary output field | Read `outputSchema`. Assess: is this a classifier? Heuristic — `outputSchema` has a single primary string field whose name matches `class` / `classification` / `label` / `category` / `intent` / `decision`, OR the system prompt enumerates fewer than 10 categories. Emit when that field has no `enum`. file = schema source, element = field name. | Add `"enum": [...]` to the field — without it, the LLM produces free-form strings the runtime can't disambiguate. |
+
+---
+
+## ToolsChecker
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `VAGUE_TOOL_DESCRIPTION` | judgment | tools | Tool description missing or too vague for the LLM to choose the tool correctly | Walk tools (normalized: `.tools[]`; agent-builder: each `resources/*/resource.json` with `.type == "tool"`). Read each `.description`. Assess: does it state purpose, side effects, and when to use vs not use — enough that the model can pick this tool over its siblings? A blank description, or boilerplate like `"Process invoice"` that names no effect or selection cue, fires; a 2-3 sentence description covering purpose + effects + usage does not. (The crude `<20-char` length check runs in the review CLI; this rule judges sufficiency.) file = tool source, element = tool name. | Write a 2-3 sentence description covering purpose, side effects, and when to use vs not use the tool. |
+| `LC_TOOL_MINIMALITY_SUFFICIENCY` | warning | tools | Tools exist that the prompt never names, OR the prompt describes responsibilities with no tool to fulfill them | Read tool list + system prompt. Assess two directions: (a) every configured tool maps to a clearly stated responsibility in the prompt; (b) every prompt-stated responsibility has at least one tool to fulfill it. Emit one finding per gap, with `description` naming the unmapped tool or unmet responsibility. file = system prompt source. | Either add the responsibility to the prompt (for orphan tools), add a tool (for orphan responsibilities), or remove the unused tool. |
+| `LC_TOOL_OVERLAP` | warning | tools | Two or more tools have substantially similar descriptions or overlapping input schemas | Read all tool descriptions + `inputSchema`s. Assess pairwise: do any two tools have descriptions a user could plausibly apply to the same request, or input schemas that look interchangeable? Emit one finding per overlapping pair, element = `<tool_a>,<tool_b>`. file = tool source. | Differentiate the tools — sharpen descriptions to make selection unambiguous, or merge if truly redundant. |
+| `LC_TOOL_DANGEROUS_COMBINATION` | warning | tools | Tool combination creates risk no individual tool creates alone (e.g., read tool + write tool with no guardrail) | Read full tool list. Assess pairwise / N-wise: do any combinations create capabilities none of the individual tools has alone? Classic example: a read tool (search, retrieve, query) + a write tool (send email, create record, call API) with no output guardrail = data-exfiltration path. Emit per dangerous combination. file = tool source, element = `<tool_a>,<tool_b>`. | Add a guardrail (e.g., output `pii_detection`, scope-restriction policy) that breaks the dangerous flow, or remove one tool. |
+| `LC_TOOL_DELEGATION_DEPTH` | warning | tools | Tool with `type: "agent"` has no prompt guidance on when to delegate vs handle directly | Read tool resources. Identify any with `.type == "agent"`. Read system prompt. Assess: does the prompt include delegation criteria? Emit when missing. file = tool source, element = tool name. | Add delegation criteria to the system prompt: when to call the sub-agent vs handle the query directly. |
+| `LC_TOOL_DESCRIPTION_MISLEADING` | info | tools | Tool description contradicts its declared `inputSchema` or `outputSchema` | For each tool, read `.description`, `.inputSchema`, `.outputSchema`. Assess: does the description claim behavior that contradicts the schemas (e.g., "returns the customer's full name" but `outputSchema` is `{type: "boolean"}`, or "takes a customer ID and email" but `inputSchema.properties` lists only `id`)? Emit per clear semantic mismatch (return type, field set, or arg count). file = tool source, element = tool name. | Reconcile the description with the actual schemas — pick the source of truth and update the other. |
+| `LC_MULTI_TOOL_NO_SEQUENCING` | warning | tools | Tools with implicit ordering exist together but the prompt establishes no order | Read tool list. Identify canonical ordered pairs: `search*` + `summarize*`; `get*` + `create*`; `validate` + `submit`. Read system prompt. Assess: does the prompt establish ordering with words like `first`, `then`, `after`, or numbered steps? Emit when tools-with-implicit-order exist together but no ordering language appears. file = system prompt source. | Add numbered steps or sequencing language to the system prompt. |
+| `LC_TOOL_RESPONSE_NOT_GROUNDED` | warning | tools | Agent has RAG-shaped retrieval but no grounding instruction in prompt or context resource | First confirm RAG shape: a tool whose output is unstructured prose intended to be cited, OR a Context Grounding index resource (`$resourceType: "context"` with `contextType: "index"`). Structured-record tools (`searchOrders`, `lookupCustomer`) do NOT trigger. When RAG-shaped, read system prompt AND every context resource's `description` / `query.description`. Assess: does grounding instruction (`"answer based only on retrieved content"` / `"cite the source"` / `"if no result, say I don't know"`) appear in either location? Emit when missing. file = system prompt or context resource. | Add a grounding instruction either to the system prompt or to the context resource's description. |
+
+---
+
+## GuardrailsChecker
+
+> **Validator name authority.** Not all validators have the same documentation status. Use only names you are confident the target layout supports:
+>
+> **SDK-documented built-ins** (confirmed by `uipath-python` SDK docs):
+>
+> | Validator | When it's needed | Valid scope | Valid stage |
+> |---|---|---|---|
+> | `pii_detection` | Agent handles personal data (names, emails, addresses, health info) | Agent, Llm, Tool | Pre or Post |
+> | `prompt_injection` | Agent accepts free-text user input | Llm only | PreExecution only |
+>
+> **Agent Builder platform-documented** (not in `uipath-python` public SDK docs):
+>
+> | Validator | When it's needed | Valid scope | Valid stage |
+> |---|---|---|---|
+> | `harmful_content` | Agent generates open-ended content | Agent, Llm, Tool | Pre or Post |
+> | `intellectual_property` | Agent generates code or text | Agent, Llm only | PostExecution only |
+> | `user_prompt_attacks` | User-facing agent, jailbreak risk | Llm only | PreExecution only |
+>
+> When recommending the three "platform-documented" types, do NOT cite them by name unless the specific validator is already present in the project's `guardrails` array or agent-builder schema. Instead say: *"Add an appropriate content-safety guardrail supported by this agent layout."*
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `LC_GUARDRAIL_PII_MISSING` | warning | guardrails | Agent processes personal data but has no `pii_detection` guardrail | Read `inputSchema`, `outputSchema`, system prompt. Assess: does the agent process personal data? Infer from field names + descriptions (names, emails, addresses, IDs, health info, financial info). When yes, check `.guardrails[]` for `pii_detection`. Emit when absent. file = `agent.json` (or normalized JSON). | Add a `pii_detection` guardrail (SDK-confirmed; cite by name). |
+| `LC_GUARDRAIL_INJECTION_MISSING` | warning | guardrails | User-facing free-text input but no `prompt_injection` guardrail | Read `inputSchema`. Assess: does it have free-text string fields users can fill with arbitrary content? When yes, check guardrails for `prompt_injection`. Emit when absent. Do NOT recommend `user_prompt_attacks` by name unless already present in the project's config — see validator authority note above. file = `agent.json`. | Add a `prompt_injection` guardrail (SDK-confirmed) or an appropriate injection-protection guardrail supported by this agent layout. |
+| `LC_GUARDRAIL_EVALS_CONSISTENCY` | judgment | guardrails | `enabledForEvals: true` may interfere with the eval set's intent | Read guardrails. For each with `enabledForEvals: true`, read the eval set. Assess: is the eval set testing guardrail behavior (intentional) or testing agent behavior (where guardrails may interfere)? Emit when intent unclear; reason in the description. file = guardrail source, element = guardrail name. | Set `enabledForEvals: false` for agent-behavior tests, or document that the eval set is specifically testing guardrail behavior. |
+
+---
+
+## GeneralChecker
+
+Covers system-prompt quality, settings fit, no-flow failure modes, and prompt-vs-schema concerns that need reasoning.
+
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+|---|---|---|---|---|---|
+| `LC_PROMPT_ROLE_DEFINITION` | warning | general | System prompt does not open with a clear role / persona statement | Read system prompt. Assess: does the opening paragraph state what the agent is and what it does? Emit when missing. file = system prompt source. | Add an opening sentence: `"You are an X that does Y."` |
+| `LC_PROMPT_WHEN_GUIDANCE` | warning | general | Prompt explains tool capabilities but not when to use each tool | Read prompt + tool list. Assess per tool: does the prompt explain *when* to use it, not just *what* it does? `"Use SearchKB when the user asks a factual question about company policy"` is good; `"SearchKB: searches the knowledge base"` is not. Emit per tool lacking WHEN guidance. file = system prompt source, element = tool name. | For each tool, add a "use this when …" sentence to the prompt. |
+| `LC_PROMPT_SCOPE_BOUNDARIES` | warning | general | Prompt does not explicitly state what the agent should NOT do | Read system prompt. Assess: does it contain explicit boundaries (`"Do not …"`, `"Only handle questions about X"`, `"If unsure, escalate"`)? Emit when absent. file = system prompt source. | Add a "Scope & Boundaries" section listing what is in and out of scope. |
+| `LC_PROMPT_OUTPUT_FORMAT` | warning | general | `outputSchema` is structured but prompt has no output-format instruction | Read `outputSchema`. Read system prompt. Assess: if the schema has specific named fields (not just a string), does the prompt instruct the agent how to populate them? Emit when missing. file = system prompt source. | Add an "Output Format" section showing the expected structure (or a sample). |
+| `LC_PROMPT_INSTRUCTION_CONFLICTS` | warning | general | Prompt contains internally contradictory instructions | Read system prompt. Assess: are there contradictions (`"Always respond in JSON"` alongside `"Write your response as a friendly paragraph"`)? Emit per contradiction. file = system prompt source. | Pick one instruction; remove or reconcile the other. |
+| `LC_PROMPT_OVERSPECIFICATION` | info | general | Prompt is so specific about exact phrases that it will break on reasonable paraphrase | Read system prompt. Assess: does it handle dozens of exact phrases that paraphrase trivially? A 3-page prompt with 47 explicit phrases is brittle. file = system prompt source. | Generalize the prompt — describe categories of input rather than enumerating phrases. |
+| `LC_PROMPT_STOPPING_CRITERIA` | warning | general | Prompt has no explicit stopping criteria for unresolvable cases | Read system prompt. Assess: is there a soft stop (`"If you cannot determine the answer after 3 attempts, respond with what you have"`)? Without one the agent relies entirely on `maxIterations`. Emit when missing. file = system prompt source. | Add an explicit stopping instruction for the "I cannot resolve this" case. |
+| `LC_SETTINGS_MAXITERATIONS_FIT` | judgment | general | `maxIterations` value inconsistent with the agent's stated purpose | Read `.settings.maxIterations` + system prompt. Assess: a simple classifier rarely needs >3-5; a complex multi-tool research agent may need 20-30. Extreme values inconsistent with the agent's purpose are flagged. Emit with reasoning. file = `agent.json` or normalized source. | Adjust `maxIterations` to match the agent's actual loop complexity. |
+| `LC_SETTINGS_TEMPERATURE_FIT` | judgment | general | `temperature` value inappropriate for task | Read `.settings.temperature` + system prompt. Assess: temperature 0 fits deterministic tasks (classification, extraction); non-zero on a classifier produces inconsistent labels. Emit with reasoning. file = `agent.json` or normalized source. | Set temperature to 0 for deterministic tasks; raise to 0.3-0.7 for conversational generation. |
+| `LC_FAILURE_HALLUCINATE_ARGS` | warning | general | Tool `inputSchema` has required fields with no descriptions, and the LLM must guess values | Walk tools. For each, read required fields. Assess: are required fields self-explanatory, or do they need descriptions to be filled correctly? Emit when the LLM would have to guess valid values. file = tool source, element = `<tool_name>:<param>`. | Add `description` to each required parameter explaining valid values and format. |
+| `LC_FAILURE_WRONG_TOOL` | warning | general | ≥2 tools have descriptions the LLM could reasonably apply to the same user request | Read all tool descriptions. Assess from a user's perspective: could any two be selected for the same request? Emit per ambiguous pair. file = tool source, element = `<tool_a>,<tool_b>`. | Sharpen descriptions to make tool selection unambiguous. |
+| `LC_FAILURE_PROMPT_INJECTION` | warning | general | Tool-output injection risk OR system-prompt-as-injection (jailbreak fragments) | Two shapes, fire on either: (a) **Tool-output injection** — a tool returns user-controlled content re-inserted into context without sanitisation, and no `prompt_injection` guardrail is active. (b) **System-prompt-as-injection** — the system prompt itself attempts to override platform safety, contains DAN-style markers (`"ignore previous"`, `"no restrictions"`, `"do anything now"`), or instructs the model to behave as another assistant (`"As ChatGPT, …"`). file = source of risk. | (a) Add `prompt_injection` guardrail and/or sanitize tool output. (b) Rewrite the system prompt to remove adversarial markers. |
+| `LC_FAILURE_IRREVERSIBLE_ACTION` | warning | general | Tools that create / delete / modify records exist alongside no escalation resource and no prompt confirmation requirement | Walk tools. Identify create/delete/modify capabilities. Read escalation resources + system prompt. Assess: is there an escalation path or prompt instruction to confirm before destructive actions? Emit when absent. file = tool source. | Add an escalation resource gating destructive actions, or add a confirmation requirement to the system prompt. |
+| `LC_FAILURE_AUTHORITY_ESCALATION` | warning | general | Agent has admin-level tools but no prompt restriction on context of use | Walk tools. Identify admin-level capabilities (process invocation, queue manipulation, entity modification). Read system prompt. Assess: does the prompt restrict the context in which these are used? Emit when unrestricted. file = system prompt source. | Restrict admin-level tools in the system prompt to specific user roles / scenarios, or add a guardrail. |
+| `LC_ESCALATION_OVERBROAD` | warning | general | Escalation description uses universal quantifiers without a narrowing condition | Read escalation resource `description`s (and channel `description`s). Assess: does it use universal quantifiers (`"always escalate"`, `"every classification"`, `"all responses"`, `"any output"`, `"regardless of intent"`) near the start AND lack a narrowing condition (`"if confidence < X"`, `"when classification is Uncertain"`, `"only for"`)? Emit when both hold. file = escalation source, element = escalation name. | Add a narrowing condition to the escalation description; if the intent really is to escalate everything, document why. |
+| `LC_PROMPT_FEW_SHOT_MISSING` | warning | general | Structured `outputSchema` (≥3 fields, or ≥2 with nested) but system prompt has no concrete output example | Read `outputSchema` (count fields, check for nested). Read system prompt. Assess: when schema is rich (≥3 fields OR ≥2 with at least one nested), does the prompt include a concrete example (`Example: {…}`, an `Output format:` block with values, in-prompt sample)? 1-2 flat fields don't trigger. Emit when missing. file = system prompt source. | Add a concrete example output to the system prompt — few-shot lifts structured-output accuracy on rich schemas. |
+| `LC_PROMPT_NEGATIVE_HEAVY` | warning | general | >50% of distinct prompt instructions are negative (`do not`, `never`, `avoid`, `must not`, `don't`) | Read system prompt. Tokenize into instructions (sentences, bullets). Assess: what fraction are negative? Emit when >50%. file = system prompt source. | Rewrite negative instructions as positive ones — LLMs follow `"always do X"` more reliably than `"never do not-X"`. |
+| `LC_USER_PROMPT_BIASED_EXAMPLE` | warning | general | User-role message ends with an example output containing a specific concrete class value, unconditioned on input | Read user-role message. Assess (must hold all three): (1) contains `"Example of correct output:"` / `"Example output:"` / `"For example:"` / `"Sample output:"` (or similar header); (2) the example body is JSON-shaped with specific concrete values for classification fields (`"classification": "Accepted"`, `"intent": "approve"`); (3) the template doesn't condition the example on the input (`"for an acceptance email, …"`, `"if the user is asking about billing, …"`). When all three hold, the LLM treats the example's class as the de-facto answer for ambiguous inputs. file = user message source. | Either remove the example, move it to the **system** prompt as one of several balanced examples, or condition it on input shape. |
+
+---
+
+## What this catalog cannot do
+
+The agent applies these rules from the project source as it is *checked into the repo*. It cannot:
+
+- Verify the LLM actually follows the prompt at runtime — prompt quality is assessed; LLM compliance can only be confirmed by running the agent.
+- Verify whether tool selection is correct at runtime — overlapping descriptions are flagged as risk; actual selection behavior requires running.
+- Verify whether guardrails fire correctly — structural configuration is checked; whether the underlying service catches a given input is a runtime question.
+- Validate live IDs (connection IDs, index names, recipient addresses) — these require API calls.

package/skills/uipath-review/references/platform/platform-resources-checklist.md

@@ -30,7 +30,7 @@ Quality checklist for UiPath Orchestrator resources — assets, queues, storage
 
 | Check | Severity | How to Verify |
 |---|---|---|
-| Assets follow consistent naming convention | Warning | `uip resource assets list <FOLDER_ID> --output json` |
+| Assets follow consistent naming convention | Warning | `uip or assets list --folder-path <FOLDER> --output json` |
 | Recommended pattern: `[Department]_[Process]_[AssetName]` | Info | Review asset names |
 | No duplicate asset names in the same folder | Warning | Check for duplicates |
 | Asset descriptions populated | Info | Check description fields |
@@ -65,7 +65,7 @@ Quality checklist for UiPath Orchestrator resources — assets, queues, storage
 
 | Check | Severity | How to Verify |
 |---|---|---|
-| Queue names follow consistent convention | Warning | `uip resource queues list <FOLDER_ID> --output json` |
+| Queue names follow consistent convention | Warning | `uip or queues list --folder-path <FOLDER> --output json` |
 | Recommended pattern: `[Department]_[Process]_QueueName` | Info | Review queue names |
 | Max retries configured (1-50, typically 3) | Warning | Check max retry setting |
 | Auto-retry enabled for application exceptions | Warning | Check auto-retry setting |

package/skills/uipath-review/references/review-workflow-guide.md

@@ -131,7 +131,7 @@ cat "<PROJECT_DIR>/project.json" | python3 -c "import json,sys; d=json.load(sys.
 
 ```bash
 # Run for EACH entry point file discovered above
-uip rpa validate --file-path "<ENTRY_FILE>" --project-dir "<PROJECT_DIR>" --output json --use-studio
+uip rpa validate --file-path "<ENTRY_FILE>" --project-dir "<PROJECT_DIR>" --output json
 ```
 
 **Step 3 — Report ALL results:**

package/skills/uipath-review/references/rpa/rpa-review-checklist.md

@@ -55,7 +55,7 @@ Comprehensive quality checklist for UiPath RPA projects — coded workflows (C#)
 
 | Check | Severity | How to Verify |
 |---|---|---|
-| All referenced packages are installable | Critical | `uip rpa packages install --use-studio` |
+| All referenced packages are installable | Critical | `uip rpa packages install` |
 | No unused packages | Warning | Workflow Analyzer rule ST-USG-010 |
 | Version constraints use proper syntax (`[1.0.0]`, `[1.0.0, 2.0.0)`) | Warning | Read project.json dependencies |
 | No package version conflicts | Critical | Check for multiple versions of same package family |
@@ -99,7 +99,7 @@ Comprehensive quality checklist for UiPath RPA projects — coded workflows (C#)
 
 | Check | Severity | How to Verify |
 |---|---|---|
-| All `.xaml` files pass validation | Critical | `uip rpa validate --file-path "<FILE>" --project-dir "<DIR>" --output json --use-studio` |
+| All `.xaml` files pass validation | Critical | `uip rpa validate --file-path "<FILE>" --project-dir "<DIR>" --output json` |
 | Expression language in all XAML files matches `project.json` setting | Critical | Check `expressionLanguage` consistency |
 | No C# syntax in VB.NET projects | Warning | If `expressionLanguage` is `VisualBasic`, grep all `.xaml` for C# patterns: `!=`, `&&`, `\|\|`, `null` (VB uses `Nothing`), `$"` (interpolated strings), `=>` (lambda), `//` (comments), `typeof()` (VB uses `GetType()`). These compile-fail or silently misbehave. |
 | No activities with default display names ("Sequence", "Assign", "If") | Warning | Workflow Analyzer rule ST-MRD-002 |

package/skills/uipath-review/references/rule-catalog-workflow.md

@@ -0,0 +1,90 @@
+# Rule Catalog — Workflow (SKILL.md Step 2.5)
+
+Step 2.5 of the review workflow. Runs **after** Step 2 (`uip agent validate` and related CLI validation) and **before** Step 3 (manual checklist review). Adds rule-ID-level findings to the report from two sources, in order:
+
+1. **The review CLI** (`uip agent review` / `uip codedagent review`) — all deterministic static checks.
+2. **The judgment catalog** (`agents-*-rules.md`) — checks that require the agent to read source and reason.
+
+## Procedure
+
+### 2.5a — Run the review CLI first
+
+Run the review command for the agent type, once, capturing JSON:
+
+| Agent type | Command |
+|---|---|
+| Low-code (`agent.json`) | `uip agent review "<PROJECT_DIR>" --output json` |
+| Coded (`main.py` + framework config) | `uip codedagent review "<PROJECT_DIR>" --output json` |
+
+Parse the JSON. Findings live under `Data.Issues[]`; each issue is:
+
+```json
+{
+  "RuleId": "LOWCODE_ENTRY_POINTS_MISSING",
+  "Category": "schema",
+  "Severity": "error",
+  "Description": "...",
+  "File": "entry-points.json",
+  "SuggestedFix": "..."
+}
+```
+
+The response also carries `Data.Verdict` (PASS/FAIL), `Data.Score`, `Data.Grade`, and `Data.Stats` — report the verdict/score if useful, but the `Issues[]` are the findings. Carry each issue into the report **verbatim**; do not re-derive, rename, or re-rank. These `RuleId`s are authoritative as the CLI emits them and are **not** duplicated in the skill catalog.
+
+If the review command is unavailable (CLI not installed, or a version without `agent review` / `codedagent review`), record one line in the report's "Rules Skipped" subsection with `reason: "uip agent review / codedagent review CLI not available"`, then continue with 2.5b.
+
+### 2.5b — Apply the judgment catalog
+
+1. **Identify which catalog files apply.** Use the detection table below.
+2. **Read each catalog file in full.**
+3. **Apply each rule's `detection_method`** (always the judgment form): read the named source material, reason about it, emit a finding when the criteria hold. Log the reasoning in the finding's `description`.
+4. **Track skipped rules.** If a rule cannot apply (`status: deferred`, missing optional file, no eval set to assess against), record `rule_id` + reason for the report's "Rules Skipped" subsection. **Never silently skip.**
+5. **Merge findings into the Step 5 report** under the "Rule Findings" subsection. Use the canonical line format:
+
+   ```
+   [<prefix><n>] `<rule_id>` — <file> — <description>. Fix: <suggested_fix>.
+   ```
+
+   where prefix is `C-D-` (Critical), `W-D-` (Warning), or `I-D-` (Info) per the severity mapping in [`rule-format.md`](rule-format.md).
+
+## Detection table
+
+Maps project signals to the judgment catalog files that must be loaded. Extend this table when adding new artifact types — do not edit SKILL.md.
+
+| Signals present | Project type | Catalog files |
+|---|---|---|
+| `agent.json` AND no `main.py` AND no `pyproject.toml` | Agent (low-code) | `agents/agents-common-rules.md` + `agents/agents-lowcode-rules.md` |
+| `pyproject.toml` + `main.py` + any framework config (`langgraph.json` / `llama_index.json` / `openai_agents.json` / `google_adk.json` / `pydantic_ai.json` / `agent_framework.json`) | Agent (coded) | `agents/agents-common-rules.md` + `agents/agents-coded-rules.md` |
+| `pyproject.toml` + `main.py` + `uipath.json[functions]` only (no framework config) | Agent (coded — Simple Function) | same as Agent (coded) |
+| `agent.json` + `pyproject.toml` + `main.py` (agent-builder coded layout) | Agent (low-code + coded) | all three: common + lowcode + coded; tag each finding with its source file |
+| `project.json` + `.xaml` / `.cs` | RPA | *(phase 2 — catalog not yet authored)* |
+| `*.flow` + `project.uiproj` with `ProjectType: "Flow"` | Flow | *(phase 2)* |
+| `.uipath/` or `app.config.json` | Coded App | *(phase 2)* |
+| None of the above with no agent signal | unknown | Skip Step 2.5; flag in the report's "Notes" section that no catalog matched. |
+
+## Why the split
+
+Deterministic checks (file presence, schema walks, counts, set-membership, regex, run-artifact analysis) are faster and byte-reproducible in code, so they live in the review CLI — the agent does not re-implement them. The catalog carries only what code cannot decide reliably: prompt quality, tool-selection ambiguity, framework fit, semantic schema/eval mismatches, whole-program dataflow. Running the CLI first means the catalog never re-litigates a deterministic finding.
+
+## Coexistence with manual checklists (Step 3)
+
+- The CLI + judgment catalog cover what can be checked mechanically or with focused judgment. The checklists in `references/<type>/<type>-review-checklist.md` cover broader semantic / contextual checks (PDD alignment, business-logic correctness, architectural fit).
+- Checklist rows that overlap with a rule are tagged like `*(rule: \`RULE_ID\`)*` — when reviewing, that rule (CLI or judgment) already covered it; do not re-flag.
+- Findings from the CLI + judgment catalog appear in the "Rule Findings" subsection; findings from manual review use the Critical / Warning / Info sections.
+
+## Determinism contract
+
+Two consecutive runs over the same project produce identical findings *for the review-CLI checks*. Judgment-catalog rules are best-effort identical — the agent should reason from the same evidence in the same order, but minor wording variation in `description` is acceptable.
+
+- Sort findings by (severity, category, rule_id, file, line) — never by discovery order.
+- Do not include timestamps in finding text.
+- Use relative paths from project root in finding `file` values; absolute paths in project metadata only.
+
+## Anti-patterns
+
+1. **Do not invent rule IDs.** If you observe a real issue covered by neither the review CLI nor the judgment catalog, surface it under the Critical / Warning / Info sections as a normal finding — do not promote it to a `rule_id`.
+2. **Do not re-rank severities.** The CLI's `Severity` and the catalog's `severity` are authoritative for `error` / `warning` / `info`. For `judgment` rows, log the reasoning when picking the report band.
+3. **Do not silently skip rules.** Every skip belongs in the report's "Rules Skipped" subsection with a reason.
+4. **Do not run the catalog before the CLI.** Run `uip agent review` / `uip codedagent review` first (2.5a), then `uip agent validate` is assumed already passed (Step 2) — so the judgment catalog focuses only on reasoning the CLI cannot do.
+5. **Do not load catalog files outside the detection table.** Loading low-code rules against a non-agent project produces false positives.
+6. **Do not re-implement deterministic checks inline.** Counts, regex, schema-presence, and set-membership are the review CLI's job. The skill itself ships no executable code.

package/skills/uipath-review/references/rule-format.md

@@ -0,0 +1,91 @@
+# Rule Catalog — Row Format
+
+Schema for every row in the `agents-*-rules.md` judgment catalogs. The catalog is the contract — the agent applies each rule by reasoning and emits findings using its `rule_id`, `severity`, and `suggested_fix`.
+
+The catalog is **judgment-only**: every rule requires the agent (itself an LLM) to read source and reason — prompt quality, tool-selection ambiguity, framework fit, semantic schema/eval mismatches. Deterministic checks (file presence, schema walks, counts, regex, run-artifact analysis) are **not** in the catalog; they run in the `uip agent review` / `uip codedagent review` CLI (SKILL.md Step 2.5a), which emits its findings in the same shape (`RuleId`, `Severity`, `Category`, `Description`, `File`, `SuggestedFix`).
+
+## Row schema
+
+Each catalog file uses one H2 section per logical checker (e.g., `## SchemaChecker`, `## ToolsChecker`). Inside each section, rules sit in one uniform table:
+
+```markdown
+| rule_id | severity | category | trigger | detection_method | suggested_fix |
+```
+
+| Column | Type | Source |
+|---|---|---|
+| `rule_id` | UPPER_SNAKE_CASE identifier in backticks | Stable contract. Never rename. |
+| `severity` | One of `error` / `warning` / `info` / `judgment` (always a single value) | Mapped at report time (see below). |
+| `category` | `evals` / `schema` / `tools` / `guardrails` / `general` / `code` / `security` / `runtime` | Drives report grouping. |
+| `trigger` | Short condition phrase | What the rule fires on. |
+| `detection_method` | The **judgment form** (see below) | The concrete evidence to read and how to reason about it. |
+| `suggested_fix` | One imperative sentence | The remediation. |
+
+## Severity mapping (catalog → report)
+
+| Catalog `severity` | Report band (from SKILL.md Step 5) | Finding ID prefix |
+|---|---|---|
+| `error` | Critical | `C-D-` |
+| `warning` | Warning | `W-D-` |
+| `info` | Info | `I-D-` |
+| `judgment` | Warning (default; agent picks Critical / Warning / Info based on contextual severity) | `W-D-` (or `C-D-` / `I-D-` when the agent escalates / de-escalates with reasoning logged in the finding's `description`) |
+
+The `-D-` infix marks the finding as rule-driven (vs `-V-` for Step 2 validation output, or no infix for manual checklist findings). Review-CLI findings carry the `RuleId` the CLI emits and are reported in the same "Rule Findings" subsection.
+
+## Detection method — the judgment form
+
+Every catalog row's `detection_method` is the judgment form:
+
+> **Judgment** — `Read <files>; assess whether <condition>; emit when <criteria>.` The agent reads the relevant source material (system prompt, tool descriptions, eval datapoints, schema, code) and applies the rule by reasoning. The `trigger` column states the rule; `detection_method` states the concrete evidence to inspect; the agent decides whether the rule fires and logs its reasoning in the finding's `description`.
+
+Deterministic forms (Glob, Read+JSON walk, Grep, Bash, count/threshold, set-membership) do **not** appear in catalog rows — those checks live in the review CLI. If a check can be made a reliable single-file regex, count, or schema walk, it belongs in the CLI, not here.
+
+## Status field (optional 7th column)
+
+A rule MAY add a `status` column for deferred or experimental rules:
+
+```markdown
+| rule_id | severity | category | trigger | detection_method | suggested_fix | status |
+```
+
+Allowed `status` values:
+
+- (omitted / blank) — active. Apply the rule.
+- `deferred` — documented for traceability; do not apply. Record in the report's "Rules Skipped" section with reason "deferred (status: deferred)".
+
+## The review CLI (deterministic findings)
+
+The agent runs the review command once per agent, capturing JSON:
+
+```bash
+uip agent review "<PROJECT_DIR>" --output json        # low-code
+uip codedagent review "<PROJECT_DIR>" --output json   # coded
+```
+
+It returns `Data.Issues[]` — deterministic findings keyed by `RuleId`, in the same severity/category/description/file/fix shape as a catalog row. The agent carries these into the report verbatim. The catalog does not list these `RuleId`s; the CLI's registry is their source of truth.
+
+## Constants section
+
+Judgment rows reference thresholds inline as soft cues (e.g. "a `<20-char` description is almost always too thin") rather than as hard constants — the agent reasons about sufficiency, not a fixed cutoff. A catalog file MAY still add a `## Constants` H2 if a kept rule genuinely needs a named threshold.
+
+## Worked examples
+
+**Judgment (prompt quality):**
+
+```markdown
+| `LC_PROMPT_ROLE_DEFINITION` | warning | general | System prompt does not open with a clear role / persona statement | Read the system prompt. Assess whether the opening paragraph states what the agent is and what it does. Emit when missing. file = system prompt source. | Add an opening sentence: `"You are an X that does Y."` |
+```
+
+**Judgment (tool sufficiency):**
+
+```markdown
+| `VAGUE_TOOL_DESCRIPTION` | judgment | tools | Tool description missing or too vague for the LLM to choose the tool correctly | Walk tools; read each `.description`. Assess: is it specific enough — purpose, side effects, when to use vs not — that the model can pick this tool over its siblings? Blank or boilerplate fires; a 2-3 sentence description does not. file = tool source, element = tool name. | Write a 2-3 sentence description covering purpose, side effects, and when to use vs not use the tool. |
+```
+
+## Reading order for the agent
+
+1. Read this file once.
+2. Read [`rule-catalog-workflow.md`](rule-catalog-workflow.md) for the Step 2.5 procedure (run the review CLI first, then the judgment catalog).
+3. Run the review CLI; capture `Data.Issues[]`.
+4. Read the catalog files indicated by the detection table for the current project type.
+5. Apply rows; emit findings using the canonical line format from SKILL.md Step 5.