agentops-accelerator 0.3.7__tar.gz → 0.3.9__tar.gz

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/.claude-plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.3.7",
+      "version": "0.3.8",
       "keywords": [
         "agentops",
         "evaluation",

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/.github/plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.3.7",
+      "version": "0.3.8",
       "keywords": [
         "agentops",
         "evaluation",

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/AGENTS.md

@@ -41,6 +41,7 @@ Public CLI contract:
 - `agentops init show [--dir PATH] [--reveal-secrets]`
 - `agentops init explain [--no-pager] [--format text|markdown|html] [--out PATH] [--open]`
 - `agentops eval analyze [--dir PATH] [--format text|markdown|json] [--out PATH]`
+- `agentops eval init [--dir PATH] [--force]`
 - `agentops eval run [--config PATH] [--baseline PATH] [--output DIR]`
 - `agentops eval promote-traces --source PATH [--out PATH] [--max-rows N] [--label-mode self-similarity|pending] [--apply]`
 - `agentops report generate [--in PATH] [--out PATH]`
@@ -325,7 +326,8 @@ Full schema:
 | `thresholds` | no | Map of metric → expression (e.g. `coherence: ">=3"`, `avg_latency_seconds: "<=30"`). Missing keys fall back to auto-defaults. |
 | `evaluators` | no | Advanced escape hatch: explicit list of evaluator names that overrides auto-selection. |
 | `project_endpoint` | no | Foundry project endpoint. Wins over `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT` when both are set. |
-| `execution` | no | `local` (default) — AgentOps runs the agent row-by-row locally. `cloud` — Foundry runs the agent + evaluators server-side (only valid for `name:version` agents). |
+| `execution` | no | `local` (default) — AgentOps runs the agent row-by-row locally. `cloud` — Foundry runs the agent + evaluators server-side (only valid for `name:version` agents). `azd` — delegates execution to `azd ai agent eval` and normalizes the emitted metrics. |
+| `eval_recipe` | no | Optional path to azd `eval.yaml` when `execution: azd`; omitted values auto-discover a single recipe. |
 | `publish` | no | When `execution: local`, set to `true` to also upload results to the Classic Foundry Evaluations panel. With `execution: cloud` publishing is implicit. |
 | `protocol` | no | URL-based agents only. `responses` (default Foundry hosted), `invocations` (Foundry hosted with raw JSON), or `http-json` (default generic HTTP). |
 | `request_field` | no | HTTP / invocations only. JSON key that carries the user prompt (default: `message`). |
@@ -333,6 +335,9 @@ Full schema:
 | `tool_calls_field` | no | HTTP / invocations only. Dot-path to extract tool calls for agent-workflow evaluators. |
 | `headers` | no | HTTP / invocations only. Static extra HTTP headers. |
 | `auth_header_env` | no | HTTP / invocations only. Environment variable that holds a Bearer token. |
+| `assert_path` | no | Optional ASSERT policy/results file or directory referenced by Doctor/evidence. AgentOps does not execute ASSERT. |
+| `acs_path` | no | Optional Agent Control Specification contract file or directory referenced by Doctor/evidence. AgentOps does not apply ACS controls. |
+| `redteam_path` | no | Optional red-team plan/results evidence path. AgentOps records metadata and never exposes payload text. |
 
 ### Agent Type Inference
 

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/CHANGELOG.md

@@ -3,7 +3,70 @@
 All notable changes to this project will be documented in this file.
 This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres to [Semantic Versioning](https://semver.org/).
 
-## [Unreleased]
+## [0.3.9] - 2026-06-08
+
+### Added
+- **AgentOps can now delegate Foundry eval execution to `azd ai agent eval`.**
+  Projects with an azd `eval.yaml` recipe can set `execution: azd` and
+  `eval_recipe: eval.yaml`; AgentOps invokes azd, normalizes emitted metrics
+  into `results.json`, binds thresholds including Rubric/custom dimensions, and
+  fails closed when configured thresholds do not map to emitted metrics.
+- **Governance evidence support for ASSERT, ACS, and red-team readiness.**
+  `agentops.yaml` can reference `assert_path`, `acs_path`, and `redteam_path`.
+  Doctor, Cockpit, and release evidence record path, SHA-256 hash, status, and
+  ACS checkpoint coverage without executing ASSERT, applying ACS controls, or
+  exposing red-team payload text.
+- **`agentops-governance` coding-agent skill.** The new skill drafts safe
+  evidence templates for ASSERT policies, ACS contracts, Guided Guardrail review
+  notes, and red-team readiness plans while explicitly refusing offensive
+  payload generation.
+
+### Fixed
+- **Prompt-agent tutorial now explicitly verifies the Travel Agent dataset path
+  after `agentops init`.** Step 7 now tells users to confirm
+  `agentops.yaml` points at `.agentops/data/travel-smoke.jsonl` and provides a
+  repair command if the wizard left the starter `.agentops/data/smoke.jsonl`.
+- **`agentops eval init` now reuses configured prompt-agent inputs and avoids
+  hidden azd prompts.** When `--dataset` is omitted, the command passes the
+  existing `agentops.yaml` dataset to `azd ai agent eval init`. It also runs azd
+  with `--no-prompt`, passes the configured Foundry project endpoint, agent
+  name, prompt file, and bootstrap model, and prints a progress line before the
+  potentially long Foundry initialization.
+- **Prompt-agent tutorial guidance now keeps azd eval recipes advanced-only.**
+  Step 10 now follows `workflow analyze` for the quickstart's AgentOps cloud
+  eval path and explains that `agentops eval init` requires a full azd AI agent
+  project context before `azd ai agent eval run` can resolve the Foundry
+  project.
+- **`agentops eval init` now prints safely on Windows terminals without Unicode
+  support.** The CLI falls back to an ASCII updated marker instead of raising a
+  `UnicodeEncodeError` on cp1252 consoles after it wires `execution: azd` and
+  `eval_recipe`.
+- **Foundry RBAC preflight now prevents the portal build-agent permission
+  block.** The prompt-agent, hosted-agent, and end-to-end tutorials plus the
+  packaged `agentops-eval` skill now grant `Foundry User` and `Cognitive
+  Services OpenAI User` on the parent AI Services account using stable role IDs.
+  This covers the Foundry UI's "You don't have permission to build agents"
+  failure as well as the evaluator chat-completions data-plane failure, while
+  still assigning the OpenAI role to Foundry/Azure AI managed identities used by
+  server-side graders.
+
+## [0.3.8] - 2026-06-04
+
+### Fixed
+- **`agentops init` now handles blank required wizard values gracefully.** If
+  the user presses Enter without an existing Foundry endpoint or agent default,
+  the wizard explains that AgentOps needs the missing value and re-prompts
+  instead of proceeding to a later persistence failure. Scripted blank flags
+  such as `--agent ""` now exit with the same friendly message and no traceback.
+- **`agentops init` no longer depends on undeclared PyYAML.** The setup wizard
+  now reads and writes `agentops.yaml` through the repository's `ruamel.yaml`
+  helpers, fixing the ugly `No module named 'yaml'` traceback seen in clean
+  installs.
+
+### Changed
+- **AgentOps brand tagline sequence now reads `Evaluate :: Ship :: Observe ::
+  Own`.** The startup/explain banner now matches the intended product story
+  order.
 
 ## [0.3.7] - 2026-06-01
 

{agentops_accelerator-0.3.7/src/agentops_accelerator.egg-info → agentops_accelerator-0.3.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentops-accelerator
-Version: 0.3.7
+Version: 0.3.9
 Summary: Release readiness gates and evidence for Microsoft Foundry agents
 License: MIT License
         
@@ -87,6 +87,8 @@ The project enables:
 - Baseline comparison for regression detection
 - Doctor checks for repo, CI/CD, telemetry, landing zones, and Foundry setup
 - Release evidence packs for promotion review
+- Optional `azd ai agent eval` execution with Rubric/custom metric binding
+- ASSERT, ACS, and red-team governance evidence references
 - Trace promotion into regression datasets
 - Cockpit navigation for AgentOps, Foundry, and Azure Monitor
 
@@ -100,9 +102,10 @@ ship/no-ship workflow.
 | Moment | Foundry / Azure does | AgentOps adds |
 |---|---|---|
 | Build and version | Foundry portal, Foundry SDK/Toolkit, `microsoft-foundry` skill, azd | Pins the exact candidate in `agentops.yaml` and generates the PR/release gate around it |
-| Evaluate and compare | Foundry Evaluations and official CI actions/extensions | Keeps datasets and thresholds in the repo, records evidence, and provides local/fallback runs for non-prompt targets |
+| Evaluate and compare | Foundry Evaluations, `azd ai agent eval`, Rubric evaluator, and official CI actions/extensions | Keeps datasets and thresholds in the repo, records evidence, normalizes azd/Rubric outputs, and provides local/fallback runs for non-prompt targets |
 | Observe and investigate | Foundry Monitor, Traces, Azure Monitor, App Insights | Surfaces deep links, telemetry readiness, Doctor findings, and Cockpit navigation |
 | Decide release | Branch protection, environments, approvals | Packages `evidence.json` / `evidence.md` for promotion review |
+| Govern controls | ASSERT, ACS, Foundry Guardrails, Foundry red-team scans | References reviewed artifacts by path/hash/status without executing or applying the external controls |
 | Improve from production | Production traces and Foundry datasets | Promotes reviewed trace learnings into regression candidates |
 
 The rhythm is simple: build and operate the agent in Foundry, keep the release
@@ -232,6 +235,7 @@ Doctor/Cockpit, and `[mcp]` for MCP.
 - `agentops --version` - show installed version.
 - `agentops init` - bootstrap config and seed data.
 - `agentops eval analyze` - check eval readiness.
+- `agentops eval init` - bootstrap an azd `eval.yaml` recipe and wire `execution: azd`.
 - `agentops eval run [--baseline PATH]` - run an evaluation.
 - `agentops eval promote-traces --source FILE [--apply]` - promote traces.
 - `agentops report generate` - regenerate `report.md`.

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/README.md

@@ -34,6 +34,8 @@ The project enables:
 - Baseline comparison for regression detection
 - Doctor checks for repo, CI/CD, telemetry, landing zones, and Foundry setup
 - Release evidence packs for promotion review
+- Optional `azd ai agent eval` execution with Rubric/custom metric binding
+- ASSERT, ACS, and red-team governance evidence references
 - Trace promotion into regression datasets
 - Cockpit navigation for AgentOps, Foundry, and Azure Monitor
 
@@ -47,9 +49,10 @@ ship/no-ship workflow.
 | Moment | Foundry / Azure does | AgentOps adds |
 |---|---|---|
 | Build and version | Foundry portal, Foundry SDK/Toolkit, `microsoft-foundry` skill, azd | Pins the exact candidate in `agentops.yaml` and generates the PR/release gate around it |
-| Evaluate and compare | Foundry Evaluations and official CI actions/extensions | Keeps datasets and thresholds in the repo, records evidence, and provides local/fallback runs for non-prompt targets |
+| Evaluate and compare | Foundry Evaluations, `azd ai agent eval`, Rubric evaluator, and official CI actions/extensions | Keeps datasets and thresholds in the repo, records evidence, normalizes azd/Rubric outputs, and provides local/fallback runs for non-prompt targets |
 | Observe and investigate | Foundry Monitor, Traces, Azure Monitor, App Insights | Surfaces deep links, telemetry readiness, Doctor findings, and Cockpit navigation |
 | Decide release | Branch protection, environments, approvals | Packages `evidence.json` / `evidence.md` for promotion review |
+| Govern controls | ASSERT, ACS, Foundry Guardrails, Foundry red-team scans | References reviewed artifacts by path/hash/status without executing or applying the external controls |
 | Improve from production | Production traces and Foundry datasets | Promotes reviewed trace learnings into regression candidates |
 
 The rhythm is simple: build and operate the agent in Foundry, keep the release
@@ -179,6 +182,7 @@ Doctor/Cockpit, and `[mcp]` for MCP.
 - `agentops --version` - show installed version.
 - `agentops init` - bootstrap config and seed data.
 - `agentops eval analyze` - check eval readiness.
+- `agentops eval init` - bootstrap an azd `eval.yaml` recipe and wire `execution: azd`.
 - `agentops eval run [--baseline PATH]` - run an evaluation.
 - `agentops eval promote-traces --source FILE [--apply]` - promote traces.
 - `agentops report generate` - regenerate `report.md`.

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/docs/concepts.md

@@ -166,6 +166,24 @@ trace-to-regression review. The standalone Microsoft Foundry AI Agent Evaluation
 Action/task remains useful for platform-native validation outside the AgentOps
 release-readiness flow.
 
+If a Foundry project already uses the public-preview `azd ai agent eval` recipe
+flow, set `execution: azd` and `eval_recipe: eval.yaml`. AgentOps delegates eval
+execution to azd, then normalizes metrics, binds thresholds, writes
+`results.json`, and fails closed for thresholds that have no emitted metric.
+Rubric evaluator dimensions are treated as first-class metric names.
+
+Governance artifacts are optional evidence inputs:
+
+```yaml
+assert_path: .assert/evaluation-policy.yaml
+acs_path: acs.yaml
+redteam_path: .agentops/governance/redteam-plan.md
+```
+
+AgentOps records path, hash, status, and ACS checkpoint coverage in Doctor,
+Cockpit, and release evidence. ASSERT execution, ACS enforcement, Guided
+Guardrail setup, and red-team scans remain in their owning tools.
+
 ## Evaluation Scenarios
 
 AgentOps auto-selects common evaluation patterns from the dataset:

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/docs/how-it-works.md

@@ -141,10 +141,12 @@ flowchart TD
     A["CLI parses args"] --> B["Load agentops.yaml"]
     B --> C["classify_agent"]
     C --> D["Pre-flight checks"]
-    D --> E["Read JSONL dataset"]
-    E --> F["Invoke target per row"]
-    F --> G["Run evaluators per row"]
-    G --> H["Aggregate metrics"]
+    D --> E{"execution mode"}
+    E -->|local/cloud| F["Read JSONL dataset"]
+    F --> G["Invoke target per row or submit Foundry cloud eval"]
+    G --> H["Run/collect evaluator scores"]
+    E -->|azd| AZD["Call azd ai agent eval using eval.yaml"]
+    AZD --> H
     H --> I["Evaluate thresholds"]
     I --> J["Write results.json + report.md"]
     J --> K["Sync .agentops/results/latest"]
@@ -164,7 +166,8 @@ fail, and `1` for runtime or configuration errors.
 AgentOps does not create a second production gate with a new exit-code
 contract. Instead, it projects the signals already produced by evals, Doctor,
 workflow files, Foundry control-plane checks, Azure Monitor, AI Landing Zone
-readiness, and trace-regression manifests into a release evidence pack:
+readiness, trace-regression manifests, and optional governance artifacts
+(ASSERT, ACS, red-team evidence indexes) into a release evidence pack:
 
 ```mermaid
 flowchart LR
@@ -175,6 +178,7 @@ flowchart LR
     PT --> M["trace-regression-manifest.json"]
     M --> P
     W["workflow generate"] --> P
+    G["ASSERT / ACS / red-team evidence paths"] --> P
     P --> J["evidence.json/evidence.md"]
 ```
 
@@ -185,6 +189,10 @@ flowchart LR
 - Trace promotion is local and review-first. `self-similarity` labels are useful
   for drift detection, not human-verified correctness; use `--label-mode pending`
   when a person must fill expected answers before the dataset can gate.
+- Governance artifacts are evidence-only references. AgentOps records path,
+  SHA-256 hash, file size, schema version when available, and ACS checkpoint
+  coverage; it does not execute ASSERT, apply ACS controls, configure Guided
+  Guardrails, or run red-team campaigns.
 
 ## Analyze / Generate Boundary
 

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/docs/tutorial-end-to-end.md

@@ -62,6 +62,24 @@ hosted endpoint itself is the per-environment artifact.
 > collisions become a real problem); dev is the gated promotion target
 > CI writes to.
 
+> **Name the Azure resources before provisioning.** When you use the Foundry
+> portal, `microsoft-foundry` skill, or Foundry Toolkit for this tutorial, specify
+> the resource group, Foundry / AI Services resource name, region, and model
+> deployment explicitly (for example `rg-agentops-travel-<your-alias>`,
+> `foundry-agentops-travel-<your-alias>`, `East US 2`, and `gpt-4o-mini`).
+> Replace `<your-alias>` with a short unique suffix when multiple people share
+> the same subscription. Resource group names are unique within a subscription;
+> Foundry / AI Services resource names should also be unique enough to avoid
+> Azure naming conflicts. Also ask the skill/tool to grant or verify `Foundry
+> User` access for your signed-in user (some portal screens still call this
+> `Azure AI User`) and `Cognitive Services OpenAI User` data-plane access for
+> your signed-in user plus any Foundry/Azure AI managed identities that will
+> call evaluator models. A single shared resource group is easiest for demos
+> because RBAC and cleanup happen once; production environments may use separate
+> resource groups per stage. For a fuller Azure baseline with networking,
+> identity, security, and operations patterns, see
+> [Azure AI Landing Zone](https://aka.ms/ailz).
+
 ## The cross-environment identity story (versioning callout)
 
 Each environment's Foundry version numbers or endpoint URLs diverge, but the
@@ -286,36 +304,52 @@ for creating agents, tools, tracing, evaluation, and red-team scans:
 https://github.com/Azure-Samples/microsoft-foundry-e2e-agent-observability-workshop/tree/2026-04-aie-europe
 ```
 
-### Grant data-plane access to your identity and Foundry managed identities
+### Grant agent-build and data-plane access to your identity and Foundry managed identities
 
 Both options above (prompt agent and hosted HTTP agent) eventually drive
 an `agentops eval run` that calls chat-completions on the AI Services
 account behind your Foundry project — either through Foundry's cloud
 graders or through the local AI-assisted evaluators. Creating a project
 through the portal assigns you `Foundry User` **only at the project
-scope**, which does not cover OpenAI data-plane actions on the parent
-account. Subscription `Owner` is also insufficient: its built-in role
-definition has `actions: ["*"]` but `dataActions: []`. Skipping this is
-what causes the eval to fail later with `PermissionDenied` on
+scope**. Creating/building agents in the Foundry UI can also require
+`Foundry User` on the parent Foundry / AI Services resource; some portal screens
+still use the previous role name, `Azure AI User`. `Foundry User` does not cover
+OpenAI data-plane actions on the parent account. Subscription `Owner` is
+also insufficient: its built-in role definition has `actions: ["*"]` but
+`dataActions: []`. Skipping the OpenAI role is what causes the eval to fail
+later with `PermissionDenied` on
 `Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/
 completions/action`.
 
-Run these assignments once per resource group that hosts a Foundry account
-you will evaluate against. Cloud evaluations run server-side and some agent
-or grader calls may authenticate as Foundry/Azure AI managed identities, not
-only as your signed-in user. Assigning the role only to your user can still
-leave graders failing with `AuthenticationError`.
+If your skill/tool already confirmed these role assignments, treat the commands
+below as a verification/fallback step. Otherwise, run these assignments once per
+AI Services account that hosts a Foundry project you will evaluate against. Cloud
+evaluations run server-side and some agent or grader calls may authenticate as
+Foundry/Azure AI managed identities, not only as your signed-in user. Assigning
+the role only to your user can still leave graders failing with
+`AuthenticationError`. Replace `<resource-group>` with the resource group you
+chose above, for example `rg-agentops-travel-<your-alias>`, and
+`<account-name>` with the parent Foundry / AI Services account name.
 
 ```powershell
 $subscriptionId = az account show --query id -o tsv
 $resourceGroup = "<resource-group>"
-$scope = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup"
+$accountName = "<account-name>"
+$accountScope = az cognitiveservices account show `
+  --resource-group $resourceGroup `
+  --name $accountName `
+  --query id -o tsv
 $userObjectId = az ad signed-in-user show --query id -o tsv
 
 az role assignment create `
   --assignee $userObjectId `
-  --role "Cognitive Services OpenAI User" `
-  --scope $scope
+  --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" `
+  --scope $accountScope
+
+az role assignment create `
+  --assignee $userObjectId `
+  --role "5e0bd9bd-7b93-4f28-af87-19fc36ad61bd" `
+  --scope $accountScope
 
 az resource list -g $resourceGroup `
   --query "[?identity.principalId!=null].identity.principalId" -o tsv |
@@ -323,8 +357,8 @@ az resource list -g $resourceGroup `
     az role assignment create `
       --assignee-object-id $_ `
       --assignee-principal-type ServicePrincipal `
-      --role "Cognitive Services OpenAI User" `
-      --scope $scope
+      --role "5e0bd9bd-7b93-4f28-af87-19fc36ad61bd" `
+      --scope $accountScope
   }
 ```
 
@@ -844,7 +878,22 @@ Use AgentOps for the repo-side follow-through:
 1. Add safety/adversarial rows to your eval dataset when there are repeatable
    cases worth gating in CI.
 2. Keep the Foundry red-team scan URL or summary with the release review.
-3. Re-run Doctor and evidence:
+   Store only safe metadata in the repo, for example
+   `.agentops/governance/redteam-plan.md`; keep raw payloads/results in the
+   approved secure system.
+3. If you use ASSERT or Agent Control Specification, add reviewed artifacts to
+   the repo or CI artifacts and point AgentOps at them:
+
+   ```yaml
+   assert_path: .assert/evaluation-policy.yaml
+   acs_path: acs.yaml
+   redteam_path: .agentops/governance/redteam-plan.md
+   ```
+
+   AgentOps records path, SHA-256 hash, status, and ACS checkpoint coverage in
+   release evidence. ASSERT execution, ACS enforcement, Guided Guardrail setup,
+   and red-team scans remain in their owning tools.
+4. Re-run Doctor and evidence:
 
 ```powershell
 agentops doctor --workspace . --evidence-pack
@@ -932,6 +981,8 @@ You are ready for a release review when:
   Insights, and Operate Ask AI was used for an aggregate summary when
   available.
 - Foundry red-team scans are linked or tracked as a release action.
+- ASSERT / ACS / red-team artifacts are represented as evidence-only
+  references when your governance process uses them.
 - Trace learnings have a path back into regression candidates.
 
 ## Where to go next

{agentops_accelerator-0.3.7 → agentops_accelerator-0.3.9}/docs/tutorial-hosted-agent-quickstart.md

@@ -143,6 +143,24 @@ FastAPI sample below emits custom OpenTelemetry spans only after you enable the
 observability step; a real Foundry Hosted Agent emits richer Foundry runtime
 spans.
 
+> **Name the Azure container resources up front.** If you use the
+> `microsoft-foundry` skill or Foundry Toolkit to create a hosted-agent project,
+> tell it the resource group, Foundry / AI Services resource name, region, and
+> model deployment you want, for example `rg-agentops-travel-<your-alias>`,
+> `foundry-agentops-travel-<your-alias>`, `East US 2`, and `gpt-4o-mini`.
+> Replace `<your-alias>` with a short unique suffix when multiple people share
+> the same subscription. Resource group names are unique within a subscription;
+> Foundry / AI Services resource names should also be unique enough to avoid
+> Azure naming conflicts. Also ask the skill/tool to grant or verify `Foundry
+> User` access for your signed-in user (some portal screens still call this
+> `Azure AI User`) and `Cognitive Services OpenAI User` data-plane access for
+> your signed-in user plus any Foundry/Azure AI managed identities that will
+> call evaluator models. For a recorded tutorial, one shared resource group is
+> easiest because RBAC and cleanup happen in one place; production teams may
+> split resource groups by environment. For a fuller Azure baseline with
+> networking, identity, security, and operations patterns, see
+> [Azure AI Landing Zone](https://aka.ms/ailz).
+
 ## 1. Create a clean workspace and install dependencies
 
 ```powershell
@@ -310,34 +328,49 @@ If the deployed endpoint needs a bearer token:
 $env:HOSTED_AGENT_TOKEN = "<token>"
 ```
 
-### Grant data-plane access to your identity and Foundry managed identities
+### Grant agent-build and data-plane access to your identity and Foundry managed identities
 
 The local AI-assisted evaluators that AgentOps runs in step 8 call
 chat-completions on the AI Services account that backs your Foundry
 project. Creating a project through the portal only assigns you
-`Foundry User` **at the project scope**, which does not cover the
-OpenAI data-plane action on the parent account. Even subscription
-`Owner` is insufficient: the built-in `Owner` role has `actions: ["*"]`
-but `dataActions: []`. Skipping this once causes the eval to fail with
-`PermissionDenied` on `Microsoft.CognitiveServices/accounts/OpenAI/
-deployments/chat/completions/action`.
-
-Run these assignments once per resource group hosting a Foundry account
-you will evaluate against. Local AI-assisted evaluators use your identity,
-while Foundry-hosted/server-side eval paths may use Azure AI managed
-identities from the same resource group. Assigning only the user can still
-leave server-side graders failing with `AuthenticationError`.
+`Foundry User` **at the project scope**. Creating/building agents in the
+Foundry UI can also require `Foundry User` on the parent Foundry / AI Services
+resource; some portal screens still use the previous role name,
+`Azure AI User`. `Foundry User` also does not cover the OpenAI data-plane action on
+the parent account. Even subscription `Owner` is insufficient: the built-in
+`Owner` role has `actions: ["*"]` but `dataActions: []`. Skipping the OpenAI
+role causes the eval to fail with `PermissionDenied` on
+`Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action`.
+
+If your skill/tool already confirmed these role assignments, treat the commands
+below as a verification/fallback step. Otherwise, run these assignments once per
+AI Services account hosting a Foundry project you will evaluate against. Local
+AI-assisted evaluators use your identity, while Foundry-hosted/server-side eval
+paths may use Azure AI managed identities from the same resource group.
+Assigning only the user can still leave server-side graders failing with
+`AuthenticationError`. Replace `<resource-group>` with the resource group you
+chose above, for example `rg-agentops-travel-<your-alias>`, and
+`<account-name>` with the parent Foundry / AI Services account name.
 
 ```powershell
 $subscriptionId = az account show --query id -o tsv
 $resourceGroup = "<resource-group>"
-$scope = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup"
+$accountName = "<account-name>"
+$accountScope = az cognitiveservices account show `
+  --resource-group $resourceGroup `
+  --name $accountName `
+  --query id -o tsv
 $userObjectId = az ad signed-in-user show --query id -o tsv
 
 az role assignment create `
   --assignee $userObjectId `
-  --role "Cognitive Services OpenAI User" `
-  --scope $scope
+  --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" `
+  --scope $accountScope
+
+az role assignment create `
+  --assignee $userObjectId `
+  --role "5e0bd9bd-7b93-4f28-af87-19fc36ad61bd" `
+  --scope $accountScope
 
 az resource list -g $resourceGroup `
   --query "[?identity.principalId!=null].identity.principalId" -o tsv |
@@ -345,8 +378,8 @@ az resource list -g $resourceGroup `
     az role assignment create `
       --assignee-object-id $_ `
       --assignee-principal-type ServicePrincipal `
-      --role "Cognitive Services OpenAI User" `
-      --scope $scope
+      --role "5e0bd9bd-7b93-4f28-af87-19fc36ad61bd" `
+      --scope $accountScope
   }
 ```
 
@@ -799,6 +832,11 @@ You are done when:
   `.agentops/release/latest/evidence.md`, and the workflow summary
   surfaces its Doctor finding summary.
 - Cockpit opens and shows the local eval history plus Doctor readiness.
+- Optional ASSERT, ACS, and red-team evidence artifacts are either absent
+  (Doctor stays silent) or wired through `assert_path`, `acs_path`, and
+  `redteam_path` in `agentops.yaml`. AgentOps cites their status/hash in release
+  evidence; it does not execute ASSERT, apply ACS controls, or run red-team
+  campaigns.
 
 ## Where to go next