agentops-accelerator 0.7.1__tar.gz → 0.8.0__tar.gz

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/.claude-plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.6.0",
+      "version": "0.8.0",
       "keywords": [
         "agentops",
         "evaluation",

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/.github/plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.6.0",
+      "version": "0.8.0",
       "keywords": [
         "agentops",
         "evaluation",

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/CHANGELOG.md

@@ -5,6 +5,17 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-07-14
+
+### Added
+- Add a read-only **Agent behavior** tab to the Foundry operations Azure Monitor
+  workbook, with versioned `AppEvents` / `customEvents` normalization, explicit
+  data and freshness states, separate invocation / evaluated-trace /
+  evaluation-event counts, scale-safe per-evaluator trends, and trace IDs for
+  Foundry investigation. Human trace annotations are the documented
+  `gen_ai.evaluation.result` producer; automated trace-evaluation event export
+  remains validation-dependent.
+
 ## [0.7.1] - 2026-07-01
 
 ### Fixed

{agentops_accelerator-0.7.1/src/agentops_accelerator.egg-info → agentops_accelerator-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentops-accelerator
-Version: 0.7.1
+Version: 0.8.0
 Summary: Release readiness gates and evidence for Microsoft Foundry agents
 License: MIT License
         
@@ -70,8 +70,8 @@ Continuous evaluation, safety testing, observability, and release readiness for
 <p align="center">
 <a href="https://pypi.org/project/agentops-accelerator/"><img alt="PyPI" src="https://img.shields.io/pypi/v/agentops-accelerator.svg?label=PyPI&color=blue"/></a>
 <a href="https://marketplace.visualstudio.com/items?itemName=AgentOpsAccelerator.agentops-accelerator"><img alt="VS Code Extension" src="https://img.shields.io/badge/VS%20Code-Extension-007ACC.svg?logo=visualstudiocode"/></a>
-<a href="https://github.com/Azure/agentops/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/Azure/agentops/actions/workflows/ci.yml/badge.svg?branch=develop"/></a>
-<a href="https://github.com/Azure/agentops/actions/workflows/release.yml"><img alt="Release" src="https://github.com/Azure/agentops/actions/workflows/release.yml/badge.svg"/></a>
+<a href="https://github.com/Azure/agentops/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/Azure/agentops/ci.yml?branch=develop&label=CI"/></a>
+<a href="https://github.com/Azure/agentops/actions/workflows/release.yml"><img alt="Release" src="https://img.shields.io/github/actions/workflow/status/Azure/agentops/release.yml?label=Release"/></a>
 <a href="https://github.com/Azure/agentops/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-green.svg"/></a>
 </p>
 

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/README.md

@@ -16,8 +16,8 @@ Continuous evaluation, safety testing, observability, and release readiness for
 <p align="center">
 <a href="https://pypi.org/project/agentops-accelerator/"><img alt="PyPI" src="https://img.shields.io/pypi/v/agentops-accelerator.svg?label=PyPI&color=blue"/></a>
 <a href="https://marketplace.visualstudio.com/items?itemName=AgentOpsAccelerator.agentops-accelerator"><img alt="VS Code Extension" src="https://img.shields.io/badge/VS%20Code-Extension-007ACC.svg?logo=visualstudiocode"/></a>
-<a href="https://github.com/Azure/agentops/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/Azure/agentops/actions/workflows/ci.yml/badge.svg?branch=develop"/></a>
-<a href="https://github.com/Azure/agentops/actions/workflows/release.yml"><img alt="Release" src="https://github.com/Azure/agentops/actions/workflows/release.yml/badge.svg"/></a>
+<a href="https://github.com/Azure/agentops/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/Azure/agentops/ci.yml?branch=develop&label=CI"/></a>
+<a href="https://github.com/Azure/agentops/actions/workflows/release.yml"><img alt="Release" src="https://img.shields.io/github/actions/workflow/status/Azure/agentops/release.yml?label=Release"/></a>
 <a href="https://github.com/Azure/agentops/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-green.svg"/></a>
 </p>
 

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/docs/evaluation.md

@@ -255,3 +255,28 @@ evaluators:
 - Use `self-similarity` only for drift detection.
 - Keep trace replay links in metadata so reviewers can investigate the original
   runtime behavior.
+
+## View Foundry trace-evaluation results in the workbook
+
+Deploy or open the Azure Monitor workbook with
+`agentops telemetry dashboard deploy` and `agentops telemetry dashboard open`,
+then select **Agent behavior**. The tab reads Microsoft Foundry-owned
+compatible `gen_ai.evaluation.result` events and observed `invoke_agent` spans
+from the selected Log Analytics workspace. Official Foundry documentation
+verifies this event for
+[human trace annotations](https://learn.microsoft.com/azure/foundry/observability/how-to/trace-annotations#log-end-user-feedback-as-trace-annotations);
+automated trace-evaluation export through the same schema is
+validation-dependent and must be proven in the target workspace. The tab shows
+data status and freshness first,
+then separate invocation, evaluated-trace, and evaluation-event counts,
+per-evaluator pass-rate / volume trends, raw scores grouped by evaluator, and
+recent trace IDs for investigation in Foundry Tracing.
+
+Foundry trace evaluation is a preview, platform-owned feature. The workbook is
+read-only: it does not schedule evaluations, change rules, add release gates, or
+replace the Foundry trace view. Filters work when environment, agent, version,
+and evaluator properties are present; missing versions are shown as **Version
+not reported**. Trace-ID evaluation and correlation do not require an emitted
+`gen_ai.agent.id`. For supported table shapes, verified producers, state
+meanings, schema assumptions, and trace-correlation instructions, see the packaged
+[workbook authoring guide](../src/agentops/templates/workbooks/README.md#agent-behavior-tab).

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/docs/how-it-works.md

@@ -20,6 +20,11 @@ references the candidate those tools produced and adds the repo-controlled
 release proof:
 config, gates, artifacts, PR reports, Doctor diagnostics, release evidence,
 trace-to-regression promotion, and Cockpit links back to Foundry/Azure Monitor.
+The additive **Agent behavior** tab in the Foundry operations Azure Monitor
+workbook reads compatible platform-owned evaluation events for cross-evaluator
+trends and trace correlation while leaving evaluation and trace drilldown in
+Foundry. Human trace annotations are a documented event producer; automated
+trace-evaluation event export remains validation-dependent.
 
 ### Key Principles
 

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/plugins/agentops/package.json

@@ -2,7 +2,7 @@
   "name": "agentops-accelerator",
   "displayName": "AgentOps Accelerator — Skills for GitHub Copilot",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.6.0",
+  "version": "0.8.0",
   "publisher": "AgentOpsAccelerator",
   "icon": "icon.png",
   "license": "MIT",

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/plugins/agentops/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agentops-accelerator",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.6.0",
+  "version": "0.8.0",
   "author": {
     "name": "AgentOps Accelerator",
     "url": "https://github.com/Azure/agentops"

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/src/agentops/agent/cockpit.py

@@ -1856,7 +1856,8 @@ def _build_open_in_foundry(
             "title": "Foundry operations dashboard",
             "description": (
                 "Azure Monitor workbook with PTU capacity, token traffic, "
-                "latency percentiles, and throttling. Deploy or open it with "
+                "latency percentiles, throttling, and read-only Foundry "
+                "trace-evaluation behavior. Deploy or open it with "
                 "`agentops telemetry dashboard`."
             ),
             "url": workbook_url,

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/src/agentops/cli/app.py

@@ -85,7 +85,7 @@ telemetry_app = typer.Typer(
 dashboard_app = typer.Typer(
     help=(
         "Deploy, open, and export the Foundry operations Azure Monitor "
-        "workbook (capacity, traffic and tokens, latency, errors)."
+        "workbook (capacity, traffic and tokens, latency, errors, agent behavior)."
     )
 )
 telemetry_app.add_typer(dashboard_app, name="dashboard")
@@ -836,10 +836,11 @@ EXPLAIN_PAGES: dict[tuple[str, ...], ExplainPage] = {
         summary=(
             "Deploys, opens, and exports the Foundry operations Azure Monitor "
             "workbook: capacity (PTU), traffic and tokens, latency percentiles, "
-            "and errors and throttling for an Azure OpenAI resource.",
+            "errors and throttling, and read-only Foundry trace-evaluation "
+            "behavior for an Azure OpenAI resource.",
             "The workbook is scoped per Azure OpenAI resource and per Log "
-            "Analytics workspace and reads from AzureMetrics and "
-            "AzureDiagnostics.",
+            "Analytics workspace and reads from AzureMetrics, AzureDiagnostics, "
+            "and Foundry-owned Application Insights events and spans.",
         ),
         children=("deploy", "open", "export"),
     ),

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/src/agentops/core/agentops_config.py

@@ -937,7 +937,6 @@ class AgentOpsConfig(BaseModel):
     protocol: Optional[Protocol] = None
     request_field: Optional[str] = None
     response_field: Optional[str] = None
-    response_fields: Dict[str, str] = Field(default_factory=dict)
     tool_calls_field: Optional[str] = None
     response_fields: Dict[str, str] = Field(
         default_factory=dict,

{agentops_accelerator-0.7.1 → agentops_accelerator-0.8.0}/src/agentops/services/azd_eval_init.py

@@ -831,6 +831,7 @@ def _azd_evaluator_selection_from_config(
     data = load_yaml(config_path)
     raw_evaluators = data.get("evaluators")
     names: list[str] = []
+    signals: list[str]
     if isinstance(raw_evaluators, list):
         for item in raw_evaluators:
             raw_name = item.get("name") if isinstance(item, dict) else item
@@ -841,7 +842,7 @@ def _azd_evaluator_selection_from_config(
             if mapped and mapped not in names:
                 names.append(mapped)
         selection_source = "explicit agentops.yaml evaluators"
-        signals = ("Using explicit evaluators from agentops.yaml.",)
+        signals = ["Using explicit evaluators from agentops.yaml."]
     else:
         selection_source = "AgentOps recommendation"
         signals = []
@@ -861,10 +862,10 @@ def _azd_evaluator_selection_from_config(
                     names.append(mapped)
         except (FileNotFoundError, OSError, ValueError) as exc:
             selection_source = "baseline fallback"
-            signals = (
+            signals = [
                 f"Could not inspect dataset for evaluator inference: {exc}",
                 "Using baseline evaluators only.",
-            )
+            ]
         if not names:
             names.extend(_DEFAULT_AZD_EVALUATORS)
     raw_rubrics = data.get("rubrics")

agentops_accelerator-0.8.0/src/agentops/templates/workbooks/README.md

@@ -0,0 +1,169 @@
+# Foundry operations dashboard (Azure Monitor workbook)
+
+`foundry-ops.workbook.json` is an Azure Monitor **gallery-template workbook** that
+visualizes Azure AI Foundry / Azure OpenAI operational metrics end to end: PTU
+utilization, PAYG spillover, traffic and token consumption, latency percentiles,
+error / throttling rates, and Foundry-owned trace-evaluation results.
+
+The workbook is scoped **per Azure OpenAI resource** and **per Log Analytics
+workspace**. Pick the subscription, workspace, and Azure OpenAI resource in the
+parameters bar, then use the deployment, model, time-range, and streaming
+filters to narrow each tab.
+
+## Tabs
+
+| Tab | Shows |
+| --- | --- |
+| Capacity | `PTU_Avg_Pct`, `PTU_Max_Pct`, `Ratelimit`, PAYG `Spillover`, `Aggregated_PTU_With_Spillover`, `PTU_Normalizado` |
+| Traffic and tokens | `TotalCalls`, `AzureOpenAIRequests`, `InputTokens`, `OutputTokens`, `TotalTokens`, `ProcessedPromptTokens`, `GeneratedTokens`, `TokensPorRequest`, and a PTU vs PAYG spillover pivot |
+| Latency | `TTFT`, `TBT`, `TTLT`, tokens/sec, time-to-response, and `P95` / `P99` / `Avg` / `Max` over `DurationMs` |
+| Errors and throttling | HTTP `429` / `400` / `500` counts, `BlockedCalls`, `TasaThrottling_Pct`, `TasaError_Pct` |
+| Agent behavior | Data status and freshness, observed `invoke_agent` invocations, evaluated traces, evaluation-event counts, per-evaluator pass rate / volume / raw-score trends, and recent failed or low-score trace correlation |
+
+`PTU_Normalizado` (`PTU% * 100000`) is a **display-only** scaling series so the
+0-100 PTU utilization percentage can share a Y axis with raw token counts in a
+single timechart. Read true utilization from `PTU_Avg_Pct` / `PTU_Max_Pct`.
+
+## KQL queries
+
+The raw KQL for each derived-metric family lives under
+[`queries/`](./queries), so operators can reuse the logic outside the workbook:
+
+| File | Derived metrics |
+| --- | --- |
+| [`capacity_ptu_spillover.kql`](./queries/capacity_ptu_spillover.kql) | PTU avg/max %, Ratelimit, Spillover, Aggregated PTU with spillover, PTU_Normalizado |
+| [`traffic_tokens.kql`](./queries/traffic_tokens.kql) | Calls, input/output/total tokens, processed/generated tokens, tokens per request, PTU vs PAYG split |
+| [`latency_percentiles.kql`](./queries/latency_percentiles.kql) | TTFT/TBT/TTLT, tokens/sec, time-to-response, and P95/P99/Avg/Max over DurationMs |
+| [`errors_throttling.kql`](./queries/errors_throttling.kql) | HTTP 429/400/500 counts, blocked calls, throttling rate, error rate |
+| [`agent_behavior.kql`](./queries/agent_behavior.kql) | Versioned `agent_behavior/v1` normalization for Foundry evaluation events and observed `invoke_agent` spans across workspace-based and classic Application Insights tables |
+
+The standalone `.kql` files expose the parameter bindings as `let` declarations
+at the top; the workbook substitutes the same values through its parameter bar.
+
+## Agent behavior tab
+
+The **Agent behavior** tab is an additive, read-only adapter over compatible
+`gen_ai.evaluation.result` events. Current official Foundry documentation
+explicitly verifies this event for
+[human trace annotations](https://learn.microsoft.com/azure/foundry/observability/how-to/trace-annotations#log-end-user-feedback-as-trace-annotations).
+It does not clearly guarantee that every automated Foundry trace-evaluation
+result is exported through the same Application Insights event shape.
+Automated event support is therefore **validation-dependent** and must be
+proven in the target workspace. Trace evaluation and annotations are preview,
+platform-owned features. AgentOps does not create, schedule, gate, or edit
+them.
+
+The `agent_behavior/v1` KQL normalizer supports both recognized Application
+Insights shapes:
+
+- workspace-based `AppEvents` (`Name`, `Properties`) and `AppDependencies`
+  (`Properties`);
+- classic `customEvents` (`name`, `customDimensions`) and `dependencies`
+  (`customDimensions`).
+
+The mapping uses an explicit set of known `gen_ai.*` properties and keeps the
+raw property bag in the normalized row. Missing optional values do not remove
+events. Missing agent versions appear as **Version not reported**, and
+nonnumeric score values remain visible as raw score text rather than becoming
+zero. Trace-ID evaluation and correlation do not require `gen_ai.agent.id`;
+agent and version filters simply use the metadata when a producer reports it.
+
+The status row appears before quality results and distinguishes these states:
+
+| State | Meaning |
+| --- | --- |
+| `Schema unavailable` | Matching event names exist, but none of the v1 evaluator, score, or label properties are recognized. Inspect the retained raw properties before updating the versioned mapping. |
+| `No access` | The workbook query shows a native permission error. Request `Log Analytics Reader` on the selected workspace. |
+| `No data` | The recognized event tables contain no compatible `gen_ai.evaluation.result` events in the selected time range. This can also mean automated trace evaluation does not export this event shape in the target environment. |
+| `Filter empty` | Events exist in the time range, but none match the environment, agent, version, and evaluator filters. |
+| `Possible ingestion delay` | The newest matching event is more than 15 minutes old. This may be ingestion delay or simply no recent Foundry trace evaluation. |
+
+The three count columns intentionally have different meanings:
+
+- **Observed invoke_agent invocations** counts distinct invocation keys visible
+  in the selected workspace.
+- **Evaluated traces** counts distinct trace IDs represented by evaluation
+  events.
+- **Evaluation events** counts evaluator results. One evaluated trace can emit
+  several events.
+
+The workbook does not calculate evaluation coverage because the workspace might
+not contain a complete invocation denominator. Pass rate uses only recognized
+pass/fail labels. Raw score trends stay in a table grouped by evaluator because
+different evaluators can use different scales; pass-rate and event-volume charts
+use comparable units.
+
+The recent-events table keeps trace, response, and conversation IDs when
+available. Copy the trace ID, open the matching Microsoft Foundry project,
+select **Tracing**, and search for the ID. The workbook cannot build a reliable
+project-specific Foundry deep link from a Log Analytics workspace ID alone.
+
+## Required diagnostic settings
+
+The Azure OpenAI resource must send both of these log categories to the Log
+Analytics workspace the workbook queries:
+
+- `RequestResponse`
+- `AzureOpenAIRequestUsage`
+
+If they are missing, create them with:
+
+```bash
+az monitor diagnostic-settings create \
+  --name agentops-foundry-ops \
+  --resource <azure-openai-resource-id> \
+  --workspace <log-analytics-workspace-id> \
+  --logs '[{"category":"RequestResponse","enabled":true},{"category":"AzureOpenAIRequestUsage","enabled":true}]'
+```
+
+`agentops doctor` flags this automatically (rule
+`waf.observability.aoai_diagnostic_categories`) and prints the exact command.
+
+## Required roles
+
+| Role | Scope | Why |
+| --- | --- | --- |
+| `Workbook Contributor` | Resource group that will hold the workbook | Deploy / update the `Microsoft.Insights/workbooks` resource |
+| `Log Analytics Reader` | The Log Analytics workspace | Let the workbook query Azure OpenAI metrics/logs and Foundry-owned Application Insights events/spans |
+
+## How to use it
+
+Ship it with the CLI:
+
+```bash
+# Emit the ARM template without touching Azure.
+agentops telemetry dashboard deploy --dry-run
+
+# Deploy the workbook (needs Workbook Contributor on the resource group).
+agentops telemetry dashboard deploy
+
+# Open the workbook in the Azure portal.
+agentops telemetry dashboard open
+
+# Copy the workbook JSON to a local path for manual import.
+agentops telemetry dashboard export --out ./foundry-ops.workbook.json
+```
+
+Or import it manually: **Azure portal → Monitor → Workbooks → New → Advanced
+Editor**, paste the contents of `foundry-ops.workbook.json`, then Apply and Save,
+selecting the target workspace and Azure OpenAI resource.
+
+## Authoring note (validate before GA)
+
+This workbook JSON was authored from the issue's KQL and the Azure Monitor
+workbook schema **without a live Azure environment**. The Agent behavior
+normalizer is validated with repository fixtures for both recognized table
+shapes, missing optional fields, nonnumeric scores, multiple evaluators, absent
+versions, and an unrecognized future schema. No live Foundry evaluation results
+were invented or claimed. These synthetic fixtures validate the adapter; they
+are not evidence that an automated Foundry trace-evaluation producer emits the
+same event schema.
+
+Before relying on the workbook in production, smoke-validate it against a real
+Log Analytics workspace that receives `RequestResponse`,
+`AzureOpenAIRequestUsage`, Foundry `invoke_agent` spans, and
+`gen_ai.evaluation.result` events. Confirm the parameters resolve, each tab
+renders, data status is accurate, trace IDs correlate in Foundry Tracing, and
+the derived metrics match expectations. Validate human-annotation and automated
+trace-evaluation producers separately. Update the versioned normalizer instead
+of silently accepting renamed properties.