agentops-accelerator 0.3.3__tar.gz → 0.3.5__tar.gz

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/.claude-plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.3.3",
+      "version": "0.3.5",
       "keywords": [
         "agentops",
         "evaluation",

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/.github/plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.3.3",
+      "version": "0.3.5",
       "keywords": [
         "agentops",
         "evaluation",

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/CHANGELOG.md

@@ -5,6 +5,58 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres
 
 ## [Unreleased]
 
+## [0.3.5] - 2026-06-01
+
+### Changed
+- **`agentops-eval` coding-agent skill now preflights the data-plane RBAC
+  step that the Foundry portal does not assign by default.** Creating a
+  Foundry project through the portal only grants the user `Foundry User`
+  at the *project* scope, which does not cover
+  `Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action`
+  on the parent AI Services account where chat completions actually live.
+  Subscription `Owner` is also insufficient because the built-in `Owner`
+  role definition has `actions: ["*"]` but `dataActions: []`. The first
+  `agentops eval run` against a fresh workspace therefore failed with
+  `PermissionDenied` on every AI-assisted evaluator and every cloud-eval
+  grader. The skill's new **Step 0.5 - Ensure data-plane RBAC on the AI
+  Services account** resolves the Foundry project endpoint from
+  `.azure/<env>/.env` or `.agentops/.env`, looks up the backing AI
+  Services account + resource group with
+  `az cognitiveservices account list`, fetches the signed-in object ID
+  with `az ad signed-in-user show`, and runs an idempotent
+  `az role assignment create` for `Cognitive Services OpenAI User` at
+  the resource-group scope before handing off to `agentops eval analyze`.
+  This keeps the skill experience consistent with the new manual
+  instructions added to the prompt-agent, hosted-agent, and end-to-end
+  tutorials, so users running the skill against a fresh Foundry project
+  no longer hit the same 401 the manual tutorials previously hid.
+
+## [0.3.4] - 2026-06-01
+
+### Fixed
+- **`agentops eval run` in local execution mode no longer fails with
+  `Missing environment variables: AZURE_OPENAI_ENDPOINT` when only the
+  Foundry project endpoint is configured.** `CONTRIBUTING.md` and the
+  user-facing env-var docs both stated that `AZURE_OPENAI_ENDPOINT` is
+  "auto-derived from the project endpoint when absent", but
+  `pipeline/runtime.py::_model_config` only read the explicit
+  `AZURE_OPENAI_ENDPOINT` env var with no fallback — so a fresh workspace
+  created by `agentops init` (which writes `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`
+  but not `AZURE_OPENAI_ENDPOINT`) would always trip the missing-env error
+  the first time AI-assisted evaluators tried to run locally. The new
+  helper `agentops.utils.azure_endpoints.derive_openai_endpoint_from_project`
+  trims the trailing `/api/projects/<name>` segment from a Foundry project
+  URL (covering both `services.ai.azure.com` and the legacy
+  `cognitiveservices.azure.com` hosts) to recover the AI Services account
+  base URL, which is exactly what the `openai` and `azure-ai-evaluation`
+  SDKs want. `_model_config` now uses the derived value as a fallback
+  whenever `AZURE_OPENAI_ENDPOINT` is unset, so the documented behavior
+  finally matches the runtime. When `AZURE_OPENAI_DEPLOYMENT` is the only
+  thing missing, the error message now points users at the deployment list
+  in the Foundry portal *and* mentions the `execution: cloud` escape hatch
+  in `agentops.yaml` so the next step is obvious without leaving the
+  terminal.
+
 ## [0.3.3] - 2026-05-31
 
 ### Changed

{agentops_accelerator-0.3.3/src/agentops_accelerator.egg-info → agentops_accelerator-0.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentops-accelerator
-Version: 0.3.3
+Version: 0.3.5
 Summary: Release readiness gates and evidence for Microsoft Foundry agents
 License: MIT License
         

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/docs/tutorial-end-to-end.md

@@ -286,6 +286,34 @@ for creating agents, tools, tracing, evaluation, and red-team scans:
 https://github.com/Azure-Samples/microsoft-foundry-e2e-agent-observability-workshop/tree/2026-04-aie-europe
 ```
 
+### Grant your identity data-plane access to the AI Services account
+
+Both options above (prompt agent and hosted HTTP agent) eventually drive
+an `agentops eval run` that calls chat-completions on the AI Services
+account behind your Foundry project — either through Foundry's cloud
+graders or through the local AI-assisted evaluators. Creating a project
+through the portal assigns you `Foundry User` **only at the project
+scope**, which does not cover OpenAI data-plane actions on the parent
+account. Subscription `Owner` is also insufficient: its built-in role
+definition has `actions: ["*"]` but `dataActions: []`. Skipping this is
+what causes the eval to fail later with `PermissionDenied` on
+`Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/
+completions/action`.
+
+Run the assignment once per resource group that hosts a Foundry account
+you will evaluate against. Replace `<your-objectId>`,
+`<subscription-id>`, and `<resource-group>` with your own values (use
+`az ad signed-in-user show --query id -o tsv` to get the object ID):
+
+```powershell
+az role assignment create `
+  --assignee <your-objectId> `
+  --role "Cognitive Services OpenAI User" `
+  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>
+```
+
+Propagation usually completes within 30–120 seconds.
+
 ## 2. Create the travel eval dataset
 
 ```powershell

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/docs/tutorial-hosted-agent-quickstart.md

@@ -310,6 +310,32 @@ If the deployed endpoint needs a bearer token:
 $env:HOSTED_AGENT_TOKEN = "<token>"
 ```
 
+### Grant your identity data-plane access to the AI Services account
+
+The local AI-assisted evaluators that AgentOps runs in step 8 call
+chat-completions on the AI Services account that backs your Foundry
+project. Creating a project through the portal only assigns you
+`Foundry User` **at the project scope**, which does not cover the
+OpenAI data-plane action on the parent account. Even subscription
+`Owner` is insufficient: the built-in `Owner` role has `actions: ["*"]`
+but `dataActions: []`. Skipping this once causes the eval to fail with
+`PermissionDenied` on `Microsoft.CognitiveServices/accounts/OpenAI/
+deployments/chat/completions/action`.
+
+Run the assignment once per resource group hosting a Foundry account
+you will evaluate against (replace `<your-objectId>`,
+`<subscription-id>`, and `<resource-group>` with your values; get the
+object ID with `az ad signed-in-user show --query id -o tsv`):
+
+```powershell
+az role assignment create `
+  --assignee <your-objectId> `
+  --role "Cognitive Services OpenAI User" `
+  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>
+```
+
+Propagation usually completes within 30–120 seconds.
+
 ## 5. Initialize AgentOps interactively
 
 ```powershell

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/docs/tutorial-prompt-agent-quickstart.md

@@ -241,6 +241,40 @@ Show me the planned changes and the resulting endpoints before applying.
 
 If the skill is not available, use Path A.
 
+### Grant your identity data-plane access to the AI Services account
+
+Creating a project through the portal only assigns you `Foundry User` **at
+the project scope**. That role does not cover the OpenAI data-plane actions
+that live on the parent AI Services *account* — the chat-completions call
+that backs every AI-assisted evaluator and every cloud-eval grader. Even
+`Owner` on the subscription is not enough: the built-in `Owner` role
+definition has `actions: ["*"]` but `dataActions: []`, so it grants full
+control plane and zero data plane on Cognitive Services accounts.
+
+Skipping this step is what causes the eval grader to fail later with::
+
+    PermissionDenied: The principal `<your-objectId>` lacks the required
+    data action `Microsoft.CognitiveServices/accounts/OpenAI/deployments/
+    chat/completions/action` to perform `POST /openai/deployments/...`
+
+Run the assignment once per resource group that hosts a Foundry account
+you will evaluate against. Replace `<your-objectId>`, `<subscription-id>`,
+and `<resource-group>` with your own values (you can get the object ID
+with `az ad signed-in-user show --query id -o tsv`):
+
+```powershell
+az role assignment create `
+  --assignee <your-objectId> `
+  --role "Cognitive Services OpenAI User" `
+  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>
+```
+
+Repeat the command with the `travel-agent-dev` resource group if the dev
+project lives in a different RG. The assignment usually propagates within
+30–120 seconds. AgentOps Doctor will detect the missing assignment in a
+future release, but until then this is a manual one-time setup step per
+new environment.
+
 ## 4. Seed `travel-agent` in the sandbox project
 
 You only author the agent in **one place**: your sandbox Foundry

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/plugins/agentops/package.json

@@ -2,7 +2,7 @@
   "name": "agentops-accelerator",
   "displayName": "AgentOps Accelerator — Skills for GitHub Copilot",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.3.3",
+  "version": "0.3.5",
   "publisher": "AgentOpsAccelerator",
   "icon": "icon.png",
   "license": "MIT",

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/plugins/agentops/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agentops-accelerator",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.3.3",
+  "version": "0.3.5",
   "author": {
     "name": "AgentOps Accelerator",
     "url": "https://github.com/Azure/agentops"

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/plugins/agentops/skills/agentops-eval/SKILL.md

@@ -25,6 +25,54 @@ with a `name:version` or URL.
    (`--project-endpoint`, `--agent`, `--dataset`, …) for non-interactive
    runs. Run `agentops init show` later to inspect the resolved config.
 
+## Step 0.5 - Ensure data-plane RBAC on the AI Services account
+
+AgentOps eval (cloud graders **and** local AI-assisted evaluators) calls
+`/openai/deployments/.../chat/completions` on the AI Services account
+that backs the Foundry project. Creating a project through the Foundry
+portal only assigns the user `Foundry User` at the *project* scope,
+which does **not** cover OpenAI data-plane actions on the parent
+account. Subscription `Owner` is also insufficient because the built-in
+`Owner` role has `actions: ["*"]` but `dataActions: []`. The first
+`agentops eval run` against a fresh workspace will otherwise fail with:
+
+```
+PermissionDenied … lacks the required data action
+'Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action'
+```
+
+Run this preflight before Step 1 - it is idempotent (Azure returns
+`RoleAssignmentExists` if already granted) and takes ~5 seconds:
+
+```bash
+# 1. Resolve the AI Services account from agentops.yaml / .azure/<env>/.env
+PROJECT_ENDPOINT=$(grep -h '^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT' .azure/*/.env .agentops/.env 2>/dev/null | tail -1 | cut -d= -f2- | tr -d '"')
+ACCOUNT_HOST=$(echo "$PROJECT_ENDPOINT" | awk -F[/:] '{print $4}')
+ACCOUNT_NAME=$(echo "$ACCOUNT_HOST" | cut -d. -f1)
+
+# 2. Resolve subscription, resource group, and signed-in object ID
+SUB_ID=$(az account show --query id -o tsv)
+RG=$(az cognitiveservices account list --subscription "$SUB_ID" --query "[?name=='$ACCOUNT_NAME'].resourceGroup | [0]" -o tsv)
+OBJ_ID=$(az ad signed-in-user show --query id -o tsv)
+
+# 3. Grant data-plane access at the RG scope (covers sandbox + future evals)
+az role assignment create \
+  --assignee "$OBJ_ID" \
+  --role "Cognitive Services OpenAI User" \
+  --scope "/subscriptions/$SUB_ID/resourceGroups/$RG"
+```
+
+PowerShell equivalent: replace `$(...)` with the PowerShell variable
+assignments shown in `docs/tutorial-prompt-agent-quickstart.md`.
+
+If the user has not run `az login` yet, do that first. If
+`az cognitiveservices account list` returns an empty RG, the AI Services
+account lives in a different subscription - ask the user which one.
+
+Skip this step only if the user explicitly says the role is already
+assigned, or if a previous `agentops eval run` succeeded against the
+same Foundry account.
+
 ## Step 1 - Analyze evaluation setup
 
 Run the deterministic local triage first:

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/src/agentops/pipeline/runtime.py

@@ -65,10 +65,22 @@ def _credential() -> Any:
 
 
 def _model_config() -> Dict[str, str]:
-    from agentops.utils.azure_endpoints import normalize_azure_openai_endpoint
+    from agentops.utils.azure_endpoints import (
+        derive_openai_endpoint_from_project,
+        normalize_azure_openai_endpoint,
+    )
 
     raw_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
     endpoint = normalize_azure_openai_endpoint(raw_endpoint)
+    if not endpoint:
+        # CONTRIBUTING.md promises ``AZURE_OPENAI_ENDPOINT`` is "auto-derived
+        # from the project endpoint when absent". The Foundry project URL
+        # already encodes the AI Services account host, so we can recover
+        # the base inference endpoint without an extra round-trip or any
+        # new wizard prompt.
+        endpoint = derive_openai_endpoint_from_project(
+            os.getenv("AZURE_AI_FOUNDRY_PROJECT_ENDPOINT")
+        )
     deployment = os.getenv("AZURE_OPENAI_DEPLOYMENT") or os.getenv(
         "AZURE_AI_MODEL_DEPLOYMENT_NAME"
     )
@@ -85,9 +97,17 @@ def _model_config() -> Dict[str, str]:
     if not deployment:
         missing.append("AZURE_OPENAI_DEPLOYMENT")
     if missing:
+        hint = ""
+        if "AZURE_OPENAI_DEPLOYMENT" in missing:
+            hint = (
+                " Set AZURE_OPENAI_DEPLOYMENT to the name of a model "
+                "deployment in your Foundry project (Models + endpoints "
+                "in the portal), or switch the run to `execution: cloud` "
+                "in agentops.yaml so Foundry runs the evaluators server-side."
+            )
         raise RuntimeError(
             "AI-assisted evaluators require an evaluator model. "
-            "Missing environment variables: " + ", ".join(missing)
+            "Missing environment variables: " + ", ".join(missing) + "." + hint
         )
 
     config: Dict[str, str] = {

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/src/agentops/templates/skills/agentops-eval/SKILL.md

@@ -25,6 +25,54 @@ with a `name:version` or URL.
    (`--project-endpoint`, `--agent`, `--dataset`, …) for non-interactive
    runs. Run `agentops init show` later to inspect the resolved config.
 
+## Step 0.5 - Ensure data-plane RBAC on the AI Services account
+
+AgentOps eval (cloud graders **and** local AI-assisted evaluators) calls
+`/openai/deployments/.../chat/completions` on the AI Services account
+that backs the Foundry project. Creating a project through the Foundry
+portal only assigns the user `Foundry User` at the *project* scope,
+which does **not** cover OpenAI data-plane actions on the parent
+account. Subscription `Owner` is also insufficient because the built-in
+`Owner` role has `actions: ["*"]` but `dataActions: []`. The first
+`agentops eval run` against a fresh workspace will otherwise fail with:
+
+```
+PermissionDenied … lacks the required data action
+'Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action'
+```
+
+Run this preflight before Step 1 - it is idempotent (Azure returns
+`RoleAssignmentExists` if already granted) and takes ~5 seconds:
+
+```bash
+# 1. Resolve the AI Services account from agentops.yaml / .azure/<env>/.env
+PROJECT_ENDPOINT=$(grep -h '^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT' .azure/*/.env .agentops/.env 2>/dev/null | tail -1 | cut -d= -f2- | tr -d '"')
+ACCOUNT_HOST=$(echo "$PROJECT_ENDPOINT" | awk -F[/:] '{print $4}')
+ACCOUNT_NAME=$(echo "$ACCOUNT_HOST" | cut -d. -f1)
+
+# 2. Resolve subscription, resource group, and signed-in object ID
+SUB_ID=$(az account show --query id -o tsv)
+RG=$(az cognitiveservices account list --subscription "$SUB_ID" --query "[?name=='$ACCOUNT_NAME'].resourceGroup | [0]" -o tsv)
+OBJ_ID=$(az ad signed-in-user show --query id -o tsv)
+
+# 3. Grant data-plane access at the RG scope (covers sandbox + future evals)
+az role assignment create \
+  --assignee "$OBJ_ID" \
+  --role "Cognitive Services OpenAI User" \
+  --scope "/subscriptions/$SUB_ID/resourceGroups/$RG"
+```
+
+PowerShell equivalent: replace `$(...)` with the PowerShell variable
+assignments shown in `docs/tutorial-prompt-agent-quickstart.md`.
+
+If the user has not run `az login` yet, do that first. If
+`az cognitiveservices account list` returns an empty RG, the AI Services
+account lives in a different subscription - ask the user which one.
+
+Skip this step only if the user explicitly says the role is already
+assigned, or if a previous `agentops eval run` succeeded against the
+same Foundry account.
+
 ## Step 1 - Analyze evaluation setup
 
 Run the deterministic local triage first:

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/src/agentops/utils/azure_endpoints.py

@@ -60,3 +60,35 @@ def normalize_azure_openai_endpoint(value: Optional[str]) -> Optional[str]:
     # Trim any straggling trailing slash so callers building paths
     # never get a doubled ``//``.
     return rewritten.rstrip("/")
+
+
+# Foundry project endpoints look like
+# ``https://<account>.services.ai.azure.com/api/projects/<project>`` (or the
+# legacy ``cognitiveservices.azure.com`` host). The Azure OpenAI inference
+# endpoint that the ``openai`` and ``azure-ai-evaluation`` SDKs want is the
+# *account* base URL — i.e. the same scheme/host with the project path
+# trimmed off. ``derive_openai_endpoint_from_project`` performs that trim so
+# users who only configure ``AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`` (the value
+# the ``agentops init`` wizard already writes) get a working AI-assisted
+# evaluator run without having to hand-author ``AZURE_OPENAI_ENDPOINT`` too.
+_PROJECT_PATH_RE = re.compile(r"/api/projects/[^/?#]+/*\Z", re.IGNORECASE)
+
+
+def derive_openai_endpoint_from_project(value: Optional[str]) -> Optional[str]:
+    """Return the Azure OpenAI base URL embedded in a Foundry project endpoint.
+
+    The function is conservative: it only rewrites URLs whose final path
+    segment matches ``/api/projects/<project>`` exactly. Anything else
+    (already-base URLs, URLs with extra path segments after the project
+    name, non-Foundry hosts) is returned untouched apart from a normalizing
+    pass through :func:`normalize_azure_openai_endpoint`. ``None`` and
+    empty strings pass through unchanged so callers can keep the
+    ``os.getenv`` ergonomic of ``derive_openai_endpoint_from_project(os.getenv(...))``.
+    """
+    if value is None:
+        return None
+    stripped = value.strip()
+    if not stripped:
+        return stripped
+    trimmed = _PROJECT_PATH_RE.sub("", stripped)
+    return normalize_azure_openai_endpoint(trimmed)

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5/src/agentops_accelerator.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentops-accelerator
-Version: 0.3.3
+Version: 0.3.5
 Summary: Release readiness gates and evidence for Microsoft Foundry agents
 License: MIT License
         

{agentops_accelerator-0.3.3 → agentops_accelerator-0.3.5}/src/agentops_accelerator.egg-info/SOURCES.txt

@@ -285,6 +285,7 @@ tests/unit/test_production_telemetry.py
 tests/unit/test_prompt_deploy.py
 tests/unit/test_release_evidence.py
 tests/unit/test_runtime_conversation.py
+tests/unit/test_runtime_model_config.py
 tests/unit/test_setup_wizard.py
 tests/unit/test_skills.py
 tests/unit/test_skills_sync.py

agentops_accelerator-0.3.5/tests/unit/test_azure_endpoints.py

@@ -0,0 +1,117 @@
+"""Tests for :mod:`agentops.utils.azure_endpoints`."""
+
+from __future__ import annotations
+
+import pytest
+
+from agentops.utils.azure_endpoints import (
+    derive_openai_endpoint_from_project,
+    normalize_azure_openai_endpoint,
+)
+
+
+@pytest.mark.parametrize(
+    "raw, expected",
+    [
+        # Already-clean values are returned untouched (apart from trailing slash).
+        ("https://x.openai.azure.com", "https://x.openai.azure.com"),
+        ("https://x.openai.azure.com/", "https://x.openai.azure.com"),
+        # The portal-style suffix is stripped.
+        ("https://x.openai.azure.com/openai/v1", "https://x.openai.azure.com"),
+        ("https://x.openai.azure.com/openai/v1/", "https://x.openai.azure.com"),
+        ("https://x.openai.azure.com/openai/v1//", "https://x.openai.azure.com"),
+        # Other well-known variants.
+        ("https://x.openai.azure.com/openai", "https://x.openai.azure.com"),
+        ("https://x.openai.azure.com/openai/", "https://x.openai.azure.com"),
+        (
+            "https://x.openai.azure.com/openai/deployments",
+            "https://x.openai.azure.com",
+        ),
+        # Case-insensitive suffix match.
+        ("https://x.openai.azure.com/OpenAI/V1", "https://x.openai.azure.com"),
+        # The AI Services proxy host is also accepted.
+        (
+            "https://aif-prj.cognitiveservices.azure.com/openai/v1",
+            "https://aif-prj.cognitiveservices.azure.com",
+        ),
+        # Whitespace is trimmed.
+        ("  https://x.openai.azure.com/openai/v1  ", "https://x.openai.azure.com"),
+    ],
+)
+def test_normalize_strips_known_suffixes(raw: str, expected: str) -> None:
+    assert normalize_azure_openai_endpoint(raw) == expected
+
+
+def test_normalize_passes_through_none() -> None:
+    assert normalize_azure_openai_endpoint(None) is None
+
+
+def test_normalize_passes_through_empty_string() -> None:
+    assert normalize_azure_openai_endpoint("") == ""
+    assert normalize_azure_openai_endpoint("   ") == ""
+
+
+def test_normalize_only_strips_at_the_end() -> None:
+    # A path containing "openai" earlier in the URL must not be touched.
+    raw = "https://proxy.example.com/openai/v1/extra"
+    assert normalize_azure_openai_endpoint(raw) == raw.rstrip("/")
+
+
+@pytest.mark.parametrize(
+    "raw, expected",
+    [
+        # Canonical Foundry project endpoint (services.ai.azure.com host).
+        (
+            "https://ai-account-xyz.services.ai.azure.com/api/projects/proj-default",
+            "https://ai-account-xyz.services.ai.azure.com",
+        ),
+        # Trailing slash on the project segment.
+        (
+            "https://ai-account-xyz.services.ai.azure.com/api/projects/proj-default/",
+            "https://ai-account-xyz.services.ai.azure.com",
+        ),
+        # Legacy cognitiveservices host.
+        (
+            "https://acct.cognitiveservices.azure.com/api/projects/p1",
+            "https://acct.cognitiveservices.azure.com",
+        ),
+        # Project names may include hyphens and digits.
+        (
+            "https://acct.services.ai.azure.com/api/projects/travel-agent-sandbox",
+            "https://acct.services.ai.azure.com",
+        ),
+        # Case-insensitive match on the ``/api/projects/`` segment.
+        (
+            "https://acct.services.ai.azure.com/API/Projects/p1",
+            "https://acct.services.ai.azure.com",
+        ),
+        # Already a base URL — passed through unchanged (apart from trailing slash).
+        (
+            "https://acct.services.ai.azure.com",
+            "https://acct.services.ai.azure.com",
+        ),
+        # Already a base URL with the OpenAI inference suffix — still normalized.
+        (
+            "https://acct.services.ai.azure.com/openai/v1",
+            "https://acct.services.ai.azure.com",
+        ),
+    ],
+)
+def test_derive_openai_endpoint_from_project(raw: str, expected: str) -> None:
+    assert derive_openai_endpoint_from_project(raw) == expected
+
+
+def test_derive_openai_endpoint_from_project_passes_through_none() -> None:
+    assert derive_openai_endpoint_from_project(None) is None
+
+
+def test_derive_openai_endpoint_from_project_passes_through_empty() -> None:
+    assert derive_openai_endpoint_from_project("") == ""
+    assert derive_openai_endpoint_from_project("   ") == ""
+
+
+def test_derive_openai_endpoint_from_project_keeps_extra_path() -> None:
+    # Only the final ``/api/projects/<name>`` segment is trimmed; an endpoint
+    # carrying additional sub-paths (e.g. an explicit agent path) is left as-is.
+    raw = "https://acct.services.ai.azure.com/api/projects/proj/agents/foo"
+    assert derive_openai_endpoint_from_project(raw) == raw.rstrip("/")

agentops_accelerator-0.3.5/tests/unit/test_runtime_model_config.py

@@ -0,0 +1,100 @@
+"""Tests for :func:`agentops.pipeline.runtime._model_config`.
+
+The covered behavior:
+
+* ``AZURE_OPENAI_ENDPOINT`` is preferred when explicitly set.
+* When absent, the endpoint is auto-derived from
+  ``AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`` — the CONTRIBUTING-stated promise
+  that ``agentops init`` already satisfies for the project endpoint.
+* When neither is available the error mentions both missing variables and
+  hints the ``execution: cloud`` escape hatch for the deployment-only case.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from agentops.pipeline.runtime import _model_config
+
+
+@pytest.fixture(autouse=True)
+def _clear_env(monkeypatch: pytest.MonkeyPatch) -> None:
+    for key in (
+        "AZURE_OPENAI_ENDPOINT",
+        "AZURE_OPENAI_DEPLOYMENT",
+        "AZURE_AI_MODEL_DEPLOYMENT_NAME",
+        "AZURE_AI_FOUNDRY_PROJECT_ENDPOINT",
+        "AZURE_OPENAI_API_VERSION",
+    ):
+        monkeypatch.delenv(key, raising=False)
+
+
+def test_explicit_openai_endpoint_wins(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://explicit.openai.azure.com")
+    monkeypatch.setenv("AZURE_OPENAI_DEPLOYMENT", "gpt-4o-mini")
+    monkeypatch.setenv(
+        "AZURE_AI_FOUNDRY_PROJECT_ENDPOINT",
+        "https://different.services.ai.azure.com/api/projects/p1",
+    )
+
+    cfg = _model_config()
+
+    assert cfg["azure_endpoint"] == "https://explicit.openai.azure.com"
+    assert cfg["azure_deployment"] == "gpt-4o-mini"
+
+
+def test_endpoint_auto_derived_from_project(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv(
+        "AZURE_AI_FOUNDRY_PROJECT_ENDPOINT",
+        "https://ai-account-xyz.services.ai.azure.com/api/projects/proj-default",
+    )
+    monkeypatch.setenv("AZURE_OPENAI_DEPLOYMENT", "gpt-4o-mini")
+
+    cfg = _model_config()
+
+    assert cfg["azure_endpoint"] == "https://ai-account-xyz.services.ai.azure.com"
+    assert cfg["azure_deployment"] == "gpt-4o-mini"
+
+
+def test_missing_deployment_still_raises_with_hint(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv(
+        "AZURE_AI_FOUNDRY_PROJECT_ENDPOINT",
+        "https://acct.services.ai.azure.com/api/projects/p1",
+    )
+
+    with pytest.raises(RuntimeError) as excinfo:
+        _model_config()
+
+    message = str(excinfo.value)
+    assert "AZURE_OPENAI_DEPLOYMENT" in message
+    assert "AZURE_OPENAI_ENDPOINT" not in message
+    # The hint nudges the user toward the cloud-execution escape hatch.
+    assert "execution: cloud" in message
+
+
+def test_missing_both_lists_endpoint_and_deployment(monkeypatch: pytest.MonkeyPatch) -> None:
+    with pytest.raises(RuntimeError) as excinfo:
+        _model_config()
+
+    message = str(excinfo.value)
+    assert "AZURE_OPENAI_ENDPOINT" in message
+    assert "AZURE_OPENAI_DEPLOYMENT" in message
+
+
+def test_default_api_version_pinned(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://x.openai.azure.com")
+    monkeypatch.setenv("AZURE_OPENAI_DEPLOYMENT", "gpt-4o-mini")
+
+    cfg = _model_config()
+
+    assert cfg["api_version"] == "2025-04-01-preview"
+
+
+def test_api_version_override(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://x.openai.azure.com")
+    monkeypatch.setenv("AZURE_OPENAI_DEPLOYMENT", "gpt-4o-mini")
+    monkeypatch.setenv("AZURE_OPENAI_API_VERSION", "2024-12-01-preview")
+
+    cfg = _model_config()
+
+    assert cfg["api_version"] == "2024-12-01-preview"