agentops-accelerator 0.3.0__py3-none-any.whl

agentops/templates/skills/agentops-config/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: agentops-config
+description: Generate or update agentops.yaml (flat 1.0 schema) for AgentOps release-readiness gates. Trigger on "configure agentops", "agentops.yaml", "set up evaluation", "what should I evaluate". Infer the agent target and dataset from the codebase; ask only when nothing can be found.
+---
+
+# AgentOps Config
+
+Generate `agentops.yaml` at the project root. This file references the agent
+candidate and dataset used to answer "can we ship it?" The flat schema has only
+a handful of fields - most projects need just `version`, `agent`, and
+`dataset`.
+
+This skill configures AgentOps release gates. It does **not** create or deploy
+Foundry agents. If the user needs to create/deploy a Prompt Agent or Hosted
+Agent first, hand off to Foundry Toolkit / the `microsoft-foundry` skill / azd,
+then return here once there is a `name:version` or URL.
+
+## Step 0 - Prerequisites
+
+1. `pip install "agentops-accelerator[foundry] @ git+https://github.com/Azure/agentops.git@main"` if `agentops` is missing.
+2. Run `agentops eval analyze` first. If it reports missing or ambiguous
+   target/dataset/scenario signals, use this skill to adapt the config.
+3. If `agentops.yaml` does not exist, run `agentops init` first. The init
+   wizard already collects the agent reference and dataset path, so
+   `agentops-config` is most useful when the user wants to **tweak** an
+   existing config (add thresholds, switch to a different agent target,
+   add HTTP auth headers, etc.) rather than create one from scratch.
+
+## Step 1 - Detect the agent target
+
+Search the codebase for the strongest signal and pick one:
+
+| Signal | `agent:` value |
+|---|---|
+| Foundry Prompt Agent ID `name:N` | `"<name>:<N>"` |
+| Foundry Hosted Agent URL `https://...services.ai.azure.com/...agents/...` | the full URL |
+| Any other HTTP endpoint your agent serves (FastAPI, Express, ACA, AKS) | the full URL |
+| Direct model use (`openai.chat.completions.create(model=...)`) with no orchestration | `"model:<deployment-name>"` |
+
+Look in: `README.md`, `main.py`/`server.py`/`app.ts`, `.agentops/.env`,
+`.env`/`.env.local`, `.azure/<env>/.env`, `infra/`, IaC outputs. If nothing is
+found, ask the user once.
+
+## Step 2 - Detect the dataset
+
+If a JSONL with rows that include `input` already exists in the repo, use
+its path. Otherwise leave the default `.agentops/data/smoke.jsonl` and
+hand off to the `agentops-dataset` skill before the first run.
+
+## Step 3 - Write agentops.yaml
+
+Minimal example:
+
+```yaml
+version: 1
+agent: "my-rag:3"
+dataset: .agentops/data/smoke.jsonl
+```
+
+HTTP/JSON example:
+
+```yaml
+version: 1
+agent: "https://my-aca-app.eastus2.azurecontainerapps.io/chat"
+dataset: .agentops/data/smoke.jsonl
+request_field: message      # default is "message"
+response_field: text         # dot-path; default is "text"
+auth_header_env: MY_API_TOKEN
+```
+
+Optional extras (only add when the user asks for them):
+
+```yaml
+thresholds:
+  coherence: ">=3"
+  groundedness: ">=3"
+  avg_latency_seconds: "<=30"
+
+# Publish results to the Foundry Evaluations panel.
+# - execution: local + publish: true  → Classic Foundry (uploads metrics)
+# - execution: cloud                  → New Foundry (server-side run;
+#                                       publish is implicit, cloud always publishes)
+execution: local
+publish: true
+# project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>"
+
+# Cloud dataset submission policy. The local JSONL remains the source of truth;
+# cloud runs sync it to Foundry Data/Datasets by default.
+dataset_sync:
+  mode: auto            # auto | foundry | inline
+  # name: agentops-smoke
+  # version: content-hash
+
+evaluators:           # rare - AgentOps auto-selects from agent + dataset
+  - name: similarity
+    threshold: ">=4"
+```
+
+## Step 4 - Validate
+
+Run `agentops eval run` once. If the config is malformed AgentOps prints a
+clear error pointing at the offending key. Adjust and re-run.
+
+## Guardrails
+
+- Do **not** add legacy keys (`bundle`, `target`, `execution`, `output`,
+  `backend`). The 1.0 schema rejects them.
+- Do **not** fabricate agent IDs, endpoint URLs, or model deployment
+  names. Ask the user when uncertain.
+- Keep the file small. Auto-selection covers most metrics.
+- Keep local JSONL canonical. For cloud runs, prefer `dataset_sync.mode: auto`
+  so AgentOps keeps Foundry Data/Datasets in sync; use `inline` only for quick
+  experiments or environments without dataset upload permission.

agentops/templates/skills/agentops-dataset/SKILL.md

@@ -0,0 +1,84 @@
+---
+name: agentops-dataset
+description: Create or extend a small JSONL dataset for AgentOps release-readiness gates. Trigger on "create dataset", "generate test data", "JSONL", "more eval rows". Infer the agent's domain from the codebase and produce realistic rows; never fabricate data when the domain is unclear.
+---
+
+# AgentOps Dataset
+
+Generate a small, realistic JSONL dataset for the agent under evaluation.
+Default location: `.agentops/data/smoke.jsonl` (referenced from
+`agentops.yaml`). These rows are repo-side release-gate inputs: keep them
+reviewable and deterministic, not a full replacement for Foundry dataset
+management.
+
+## Step 0 - Prerequisites
+
+1. `pip install "agentops-accelerator @ git+https://github.com/Azure/agentops.git@main"` if `agentops` is missing.
+2. Run `agentops eval analyze` first. If it reports missing dataset columns or
+   recommends `agentops-dataset`, use this skill before the first eval run.
+3. If `agentops.yaml` does not exist, run `agentops init` first (the init
+   wizard will prompt for the agent reference, project endpoint, and
+   dataset path, then create a starter `.agentops/data/smoke.jsonl`).
+
+## Step 1 - Pick the columns
+
+Read `agentops.yaml` (and the agent code) to figure out the agent type,
+then choose the row schema:
+
+| Agent type | Required columns | Optional columns |
+|---|---|---|
+| Direct model / Q&A | `input`, `expected` | - |
+| RAG | `input`, `expected`, `context` | - |
+| Conversational | `input`, `expected` | - |
+| Tool-using agent | `input`, `expected`, `tool_calls` | `tool_definitions` |
+
+`input` is always the user prompt. `expected` is the gold answer.
+`context` is the retrieved passage(s). `tool_calls` is a list of
+`{name, arguments}` describing the expected tool invocations.
+
+## Step 2 - Ground the rows in the codebase
+
+- Read the README, system prompt, tool definitions, and any sample
+  fixtures.
+- Generate **5–10 rows** that exercise the agent's actual capabilities.
+- If the domain is unclear, generate a tiny generic draft and clearly
+  flag it as a placeholder.
+
+## Step 3 - Write the JSONL
+
+One JSON object per line, no trailing commas, UTF-8:
+
+```json
+{"input": "What is the refund policy?", "expected": "Refunds within 30 days...", "context": "Refund policy: ..."}
+```
+
+Save to the path referenced by `dataset:` in `agentops.yaml` (default
+`.agentops/data/smoke.jsonl`).
+
+This file is the AgentOps source of truth. In Foundry cloud evaluation,
+AgentOps syncs it to a stable Foundry dataset version by default and reuses the
+same Foundry dataset version while the JSONL content is unchanged. If the user
+forces `dataset_sync.mode: inline`, Foundry may show generated `eval-data-*`
+backing assets in the project Data/Datasets page.
+
+## Step 4 - Sanity-check
+
+Run a quick eval and confirm rows are picked up:
+
+```bash
+agentops eval run
+```
+
+Open `.agentops/results/latest/report.md` and confirm the row count
+matches.
+
+## Guardrails
+
+- Do not invent customer data, real names, or sensitive content.
+- Keep rows short - datasets are meant to be quick gates, not full QA
+  suites.
+- If the user already has a domain dataset, prefer pointing
+  `agentops.yaml` at that file rather than generating new rows.
+- If the user asks why Foundry shows `eval-data-*`, explain that those are
+  cloud-eval backing assets from inline compatibility mode; normal cloud runs
+  should use the stable `agentops-*` Foundry dataset.

agentops/templates/skills/agentops-eval/SKILL.md

@@ -0,0 +1,189 @@
+---
+name: agentops-eval
+description: Run AgentOps release-readiness evaluations against Foundry prompt agents, Foundry hosted endpoints, HTTP/JSON agents, or raw model deployments. Trigger on phrases like "run eval", "evaluate my agent", "benchmark", "agentops eval", "compare runs", "can we ship". Uses the flat agentops.yaml schema.
+---
+
+# AgentOps Eval
+
+End-to-end release-gate workflow: install -> init -> configure -> run -> read
+report -> decide whether the candidate is ready to ship.
+
+AgentOps evaluates an existing candidate. It does **not** create or deploy
+Foundry agents. If the user still needs a Prompt Agent or Hosted Agent, hand off
+to Foundry Toolkit / the `microsoft-foundry` skill / azd first, then come back
+with a `name:version` or URL.
+
+## Step 0 - Setup
+
+1. Install if missing: `pip install "agentops-accelerator[foundry] @ git+https://github.com/Azure/agentops.git@main"`.
+2. If `agentops.yaml` does not exist at the project root, run `agentops init`.
+   The init wizard prompts (azd-style) for the Foundry project endpoint,
+   agent reference, and dataset path, persists each answer to
+   `.agentops/.env` + `agentops.yaml` as it goes. Existing azd workspaces, or
+   runs with `--azd-env`, use `.azure/<env>/.env` instead. Pass `--no-prompt`
+   plus the explicit flags
+   (`--project-endpoint`, `--agent`, `--dataset`, …) for non-interactive
+   runs. Run `agentops init show` later to inspect the resolved config.
+
+## Step 1 - Analyze evaluation setup
+
+Run the deterministic local triage first:
+
+```bash
+agentops eval analyze
+```
+
+Use its output to decide whether the repo is ready for `agentops eval run` or
+needs skill-assisted setup. If it recommends `agentops-config`, fix the target
+and protocol. If it recommends `agentops-dataset`, create/map realistic JSONL
+rows. If it recommends `agentops-eval`, inspect the app scenario and evaluator
+expectations before running.
+
+## Step 2 - Identify the agent target
+
+Read the codebase (README, entry point, env vars) and pick the right value
+for the `agent:` field of `agentops.yaml`:
+
+| Pattern in code / env | `agent:` value |
+|---|---|
+| Foundry Prompt Agent ID like `name:1` | `"<name>:<version>"` |
+| Foundry Hosted Agent endpoint URL ending in `/agents/...` | `"https://<resource>.services.ai.azure.com/api/projects/<p>/agents/..."` |
+| Plain HTTP/JSON endpoint (FastAPI, Express, ACA, AKS) | `"https://<host>/<path>"` |
+| Raw Foundry/Azure OpenAI model deployment | `"model:<deployment-name>"` |
+
+If nothing is found, ask the user once for the agent identifier.
+
+## Step 3 - Make sure the dataset exists
+
+`agentops.yaml` points to a JSONL file (default
+`.agentops/data/smoke.jsonl`). Each row needs at least `input` and a label
+that maps to the metric you care about (`expected`, `context`,
+`tool_calls`...). If the dataset is empty or unrelated, run the
+`agentops-dataset` skill before running the eval.
+
+## Step 4 - Run the evaluation
+
+```bash
+agentops eval run
+```
+
+Optional flags:
+
+- `--config <path>` - point at a different `agentops.yaml`.
+- `--output <dir>` - choose where to write `results.json` and `report.md`
+  (defaults to `.agentops/results/<timestamp>/`).
+
+Exit codes:
+
+- `0` - succeeded and all thresholds passed
+- `2` - succeeded but at least one threshold failed (gate-friendly)
+- `1` - runtime/configuration error
+
+## Step 4b - Pick the right execution path
+
+| Target | Foundry server-side eval through AgentOps | AgentOps local runner | Default guidance |
+|---|---|---|---|
+| Foundry Prompt Agent (`name:version`) | `execution: cloud` | yes | Use cloud when the user wants the official Foundry-hosted eval record; use local for fast feedback or fallback. |
+| Foundry Hosted Agent URL | no | yes | Use local runner; optionally set `publish: true` to upload local metrics to Classic Foundry. |
+| Generic HTTP/JSON endpoint | no | yes | Use local runner. |
+| Raw model deployment | no | yes | Use local runner. |
+
+For prompt-agent CI gates, prefer AgentOps cloud eval because Foundry executes
+the managed eval while AgentOps enforces thresholds and writes normalized
+`results.json` / `report.md` artifacts. The official AI Agent Evaluation GitHub
+Action or Azure DevOps extension is still useful for standalone platform-native
+validation, but do not substitute it for the AgentOps PR gate when the user needs
+threshold enforcement, baselines, Doctor readiness, release evidence, or local
+fallback.
+
+## Step 5 - Inspect results and release evidence
+
+```bash
+agentops report generate                   # regenerate report.md from latest results.json
+agentops report generate --in <results.json>
+```
+
+Open `.agentops/results/latest/report.md`. To compare two runs, hand both
+`results.json` files to the user or run the next eval with
+`--baseline <previous-results.json>` so AgentOps adds a **Comparison vs
+Baseline** section to the report.
+
+For production promotion, generate the Doctor evidence pack:
+
+```bash
+agentops doctor --evidence-pack
+```
+
+Open `.agentops/release/latest/evidence.md`. It summarizes eval, baseline,
+Doctor, workflow, Foundry, monitoring, AI Landing Zone, and trace-regression
+readiness without creating a second exit-code contract.
+
+## Step 5b - (Optional) Promote reviewed traces to regression rows
+
+If the user has exported Foundry/App Insights traces, preview candidate
+regression rows first:
+
+```bash
+agentops eval promote-traces --source <traces.jsonl>
+```
+
+Only write files after review:
+
+```bash
+agentops eval promote-traces --source <traces.jsonl> --apply
+```
+
+Default `self-similarity` labels are for drift detection, not human-verified
+ground truth. Use `--label-mode pending` when reviewers must fill expected
+answers before the dataset gates releases.
+
+## Step 6 - (Optional) Foundry execution / visibility
+
+Two modes are supported. Both write a deep-link into
+`.agentops/results/latest/cloud_evaluation.json` and require
+`AZURE_AI_FOUNDRY_PROJECT_ENDPOINT` (or the inline `project_endpoint`).
+
+**Classic Foundry Evaluations panel** (works for any target kind):
+AgentOps runs locally first, then uploads the metrics it computed.
+
+```yaml
+execution: local
+publish: true
+# project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>"
+```
+
+**New Foundry Evaluations panel** (preview): Foundry runs the agent +
+evaluators server-side via the OpenAI Evals API. Only works for
+`name:version` Foundry agents. `publish` is implicit - a cloud run is
+always recorded by Foundry. The local JSONL remains the dataset source of
+truth; AgentOps syncs it to Foundry Data/Datasets by default and uses that
+dataset version in the Evals run.
+
+```yaml
+execution: cloud
+# project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>"
+dataset_sync:
+  mode: auto            # sync local JSONL to Foundry Data/Datasets
+```
+
+With `execution: local` and no `publish: true`, AgentOps runs locally
+and only writes local artifacts.
+
+After a cloud run, inspect `.agentops/results/latest/cloud_evaluation.json`.
+Its `dataset` block explains whether the run used inline rows or a Foundry
+dataset reference.
+
+## Tips
+
+- Evaluators are auto-selected from the agent type and dataset columns.
+  Override only when needed via the `evaluators:` block - most users do
+  not need it.
+- Set thresholds in `thresholds:` to gate CI:
+  ```yaml
+  thresholds:
+    coherence: ">=3"
+    avg_latency_seconds: "<=10"
+  ```
+- For HTTP/JSON agents that need auth, set
+  `auth_header_env: MY_TOKEN_VAR` and AgentOps adds
+  `Authorization: Bearer $MY_TOKEN_VAR`.

agentops/templates/skills/agentops-report/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: agentops-report
+description: Read, regenerate, and explain AgentOps release-gate reports. Trigger on "show report", "explain scores", "regenerate report", "what do these metrics mean", "where is the proof". Operates on results.json and report.md produced by `agentops eval run`.
+---
+
+# AgentOps Report
+
+Help the user understand a finished AgentOps run and the evidence it provides
+for the release decision. Reports explain the repo-side gate; they do not
+replace Foundry Evaluations, Traces, or Monitor drilldown.
+
+## Step 0 - Locate the run
+
+Latest run: `.agentops/results/latest/`. Each run produces:
+
+- `results.json` - machine-readable metrics, per-row scores, thresholds.
+- `report.md` - human-readable summary suitable for PR comments.
+- `cloud_evaluation.json` (when Foundry visibility is enabled) - deep-link
+  to the Foundry Evaluations panel. `mode: classic` when `execution: local`
+  and `publish: true` upload metrics to Classic Foundry; `mode: cloud` when
+  `execution: cloud` runs server-side via the OpenAI Evals API.
+
+## Step 1 - Regenerate report.md if needed
+
+```bash
+agentops report generate                   # uses .agentops/results/latest/results.json
+agentops report generate --in <results.json> --out <report.md>
+```
+
+`report generate` always reads the flat 1.0 results schema and emits
+Markdown. There is no HTML format.
+
+## Step 2 - Explain the metrics
+
+Common metrics and their meaning:
+
+| Metric | Range | Higher is better? | Notes |
+|---|---|---|---|
+| `similarity` | 1-5 | yes | LLM-judged similarity to `expected`. |
+| `coherence` | 1-5 | yes | Answer is internally consistent. |
+| `fluency` | 1-5 | yes | Natural language quality. |
+| `groundedness` | 1-5 | yes | Answer is supported by `context` (RAG). |
+| `relevance` | 1-5 | yes | Answer is on-topic for `input`. |
+| `f1_score` | 0-1 | yes | Token overlap with `expected`. |
+| `tool_call_accuracy` | 0-1 | yes | Predicted tool calls match `tool_calls`. |
+| `intent_resolution` | 0-1 | yes | User intent was resolved. |
+| `task_completion` | 0-1 | yes | Multi-step task finished. |
+| `avg_latency_seconds` | seconds | no | Wall-clock latency per row. |
+
+Pass/fail rows are derived from `thresholds:` in `agentops.yaml`. The
+exit code of the original run reflects the gate:
+
+- `0` → all thresholds passed
+- `2` → one or more thresholds failed
+- `1` → runtime error
+
+## Step 3 - Help the user act on results
+
+- For low scores on a specific metric, point at the lowest-scoring rows
+  in `results.json` (`row_metrics[]` and `item_evaluations[]`) and
+  suggest concrete prompt or retrieval changes.
+- For latency regressions, look at `run_metrics.avg_latency_seconds` and
+  per-row latency.
+- To compare a new run against a previous one, re-run with
+  `agentops eval run --baseline <previous-results.json>` and explain the
+  generated **Comparison vs Baseline** section.
+
+## Guardrails
+
+- Never invent metric values. If a metric is absent, say so.
+- Do not edit `results.json` by hand - re-run the eval.