agentops-accelerator 0.4.4__tar.gz → 0.4.5__tar.gz

{agentops_accelerator-0.4.4 → agentops_accelerator-0.4.5}/CHANGELOG.md

@@ -5,6 +5,32 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres
 
 ## [Unreleased]
 
+## [0.4.5] - 2026-06-19
+
+### Added
+- **Governance gates for HTTP agents (ASSERT and Red Team).** `agentops assert
+  run` and `agentops redteam run` now work against a live HTTP orchestrator
+  endpoint, not only model/deployment targets. Red Team wraps the HTTP endpoint
+  as an SDK-compatible target and reuses the AgentOps HTTP mapping
+  (`request_field`, `response_mode`, `stream`, custom headers). ASSERT resolves
+  `assert-ai` inside the active virtual environment, accepts non-secret values
+  from `assert.env`, can request an AAD token from the Azure CLI for local
+  auth-disabled Azure AI resources, injects the GPT-5 `max_completion_tokens`
+  shim only when configured, and materializes a runtime ASSERT config so
+  committed configs no longer need absolute artifact paths.
+- **Generated workflows run the ASSERT and Red Team gates.** `agentops workflow
+  generate` now installs the optional ASSERT/Red Team dependencies, runs those
+  gates when `assert:` or `redteam:` is present in `agentops.yaml`, uploads
+  their artifacts, and emits the corrected Red Team command quoting.
+
+### Fixed
+- **Reasoning-model judges no longer fail the eval gate in CI.** The generated
+  GitHub Actions and Azure DevOps eval and Red Team steps now forward
+  `AZURE_OPENAI_MODEL_NAME`, so AgentOps detects reasoning models (such as
+  `gpt-5-nano`) and uses `max_completion_tokens` instead of `max_tokens`. This
+  removes the judge `400` error that could break the eval gate when the judge
+  deployment is a reasoning model.
+
 ## [0.4.4] - 2026-06-18
 
 ### Added
@@ -56,6 +82,22 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres
   `eval.yaml`, so users can see why those evaluators were chosen.
   ([#323](https://github.com/Azure/agentops/issues/323))
 
+## [0.4.2] - 2026-06-17
+
+### Fixed
+- **`agentops eval init` now works with both old and new `azure.ai.agents` azd
+  extensions.** Version 0.1.40 of the extension renamed the eval subcommand from
+  `azd ai agent eval init` to `azd ai agent eval generate`, which made
+  `agentops eval init` hard-fail with `Command "init" is deprecated, use 'azd ai
+  agent eval generate' instead`. AgentOps now invokes `generate` first and
+  transparently falls back to the legacy `init` subcommand when an older
+  extension does not recognise `generate`. The fallback only triggers on
+  subcommand-name/deprecation errors; genuine failures (authentication, project
+  endpoint, timeouts) are still surfaced immediately and unchanged. All
+  previously passed flags (`--project-endpoint`, `--agent`,
+  `--gen-instruction-file`, `--eval-model`, `--dataset`, `--evaluator`) and the
+  recipe discovery/persistence behaviour are preserved.
+
 ## [0.4.1] - 2026-06-15
 
 ### Changed

{agentops_accelerator-0.4.4/src/agentops_accelerator.egg-info → agentops_accelerator-0.4.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentops-accelerator
-Version: 0.4.4
+Version: 0.4.5
 Summary: Release readiness gates and evidence for Microsoft Foundry agents
 License: MIT License
         

{agentops_accelerator-0.4.4 → agentops_accelerator-0.4.5}/plugins/agentops/skills/agentops-governance/SKILL.md

@@ -206,6 +206,115 @@ assert-ai init
 It walks them through behavior description, target callable / model /
 endpoint, dimensions, and judge presets, and writes a validated YAML.
 
+### HTTP orchestrator ASSERT
+
+If `agentops.yaml` uses `protocol: http-json` or the user says the target is an
+HTTP orchestrator, do not use ASSERT native endpoint mode. `assert-ai 0.1.0`
+posts `message/history` and expects `response`; AgentOps HTTP targets may use
+custom fields like `ask` and streamed text. Scaffold a callable adapter instead.
+
+Create `.agentops/assert_http_adapter.py`:
+
+```python
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from agentops.core.config_loader import load_agentops_config
+from agentops.pipeline.invocations import (
+    _aggregate_stream,
+    _dot_path,
+    _http_request_json,
+    _http_request_stream,
+)
+
+
+def target(message: str, history: list[dict[str, Any]] | None = None) -> str:
+    del history
+    config = load_agentops_config(Path("agentops.yaml"))
+    if not config.agent:
+        raise RuntimeError("agentops.yaml must define a top-level HTTP agent endpoint")
+
+    request_field = config.request_field or "message"
+    headers = dict(config.headers)
+    headers.setdefault("Content-Type", "application/json")
+    body = {request_field: message}
+
+    if config.response_mode in ("sse", "text"):
+        raw_body = _http_request_stream(
+            method="POST",
+            url=config.agent,
+            headers=headers,
+            body=body,
+            timeout=120,
+        )
+        return _aggregate_stream(config.response_mode, raw_body, config.stream).strip()
+
+    payload = _http_request_json(
+        method="POST",
+        url=config.agent,
+        headers=headers,
+        body=body,
+        timeout=120,
+    )
+    response_path = config.response_field or "text"
+    response_text = _dot_path(payload, response_path)
+    if response_text is None and isinstance(payload, dict):
+        for fallback in ("response", "output", "content", "message", "text"):
+            response_text = payload.get(fallback)
+            if response_text:
+                break
+    return (
+        response_text
+        if isinstance(response_text, str)
+        else json.dumps(response_text or "", ensure_ascii=False)
+    )
+```
+
+Create an ASSERT smoke from a known-good eval dataset row, not a random general
+question. For the HTTP tutorial, use:
+
+```yaml
+suite: gpt-rag-http-smoke
+run: local-http-contract-smoke
+
+default_model:
+  name: azure/chat
+
+pipeline:
+  systematize:
+    enabled: false
+  test_set:
+    enabled: false
+  inference:
+    test_set_path: test_set.jsonl
+    target:
+      callable: assert_http_adapter:target
+    max_turns: 1
+  judge:
+    taxonomy_path: taxonomy.json
+    preset:
+      - grounding
+```
+
+Append this `assert:` block to `agentops.yaml`. Discover `AZURE_API_BASE` from
+the Azure AI/OpenAI resource and set `AZURE_API_VERSION` to the version used by
+the deployment. These are not secrets. If local auth is disabled, AgentOps will
+use the signed-in Azure CLI token for the ASSERT subprocess.
+
+```yaml
+assert:
+  config: ./assert/eval_config.yaml
+  fail_on_violations: true
+  env:
+    AZURE_API_BASE: https://<azure-ai-resource>.cognitiveservices.azure.com/
+    AZURE_API_VERSION: 2024-12-01-preview
+    AGENTOPS_ASSERT_AZURE_MAX_COMPLETION_TOKENS: "true"
+    PYTHONPATH: .agentops
+```
+
 **3. Append the `assert:` block to `agentops.yaml`** (preserve every existing
 key — read the file, append the block if missing, write back):
 

{agentops_accelerator-0.4.4 → agentops_accelerator-0.4.5}/src/agentops/cli/app.py

@@ -14,13 +14,16 @@ from html import escape as html_escape
 from pathlib import Path
 from textwrap import wrap
 from collections.abc import Sequence
-from typing import Annotated, Any, Optional
+from typing import Annotated, Any, Optional, TYPE_CHECKING
 
 import typer
 
 from agentops.utils.colors import style
 from agentops.utils.logging import get_logger, setup_logging
 
+if TYPE_CHECKING:
+    from agentops.core.agentops_config import AgentOpsConfig
+
 app = typer.Typer(
     name="agentops",
     help="AgentOps - standardized evaluation workflows for AI projects.",
@@ -1574,11 +1577,13 @@ def cmd_init(
     from agentops.services.setup_wizard import (
         AGENT_TITLE,
         DATASET_TITLE,
+        ENDPOINT_SOURCE_AZD_RESOURCE_DISCOVERY,
         PROJECT_ENDPOINT_TITLE,
         REQUIRED_CONFIGURATION_MESSAGE,
         WizardAnswers,
         apply_answers,
         discover_defaults,
+        is_placeholder_agent,
         run_wizard,
         validate_agent,
         validate_dataset,
@@ -1763,12 +1768,14 @@ def cmd_init(
         force_prompt_fields = {"agent", "dataset"} if config_seeded_this_run else set()
         prompt_values = [
             defaults.project_endpoint,
-            defaults.agent,
+            None if is_placeholder_agent(defaults.agent) else defaults.agent,
             defaults.dataset,
         ]
-        will_prompt = reconfigure or bool(force_prompt_fields) or any(
-            v is None or not str(v).strip()
-            for v in prompt_values
+        will_prompt = (
+            reconfigure
+            or bool(force_prompt_fields)
+            or any(v is None or not str(v).strip() for v in prompt_values)
+            or defaults.project_endpoint_source == ENDPOINT_SOURCE_AZD_RESOURCE_DISCOVERY
         )
         if will_prompt:
             typer.echo(style("Press Enter to accept the value in brackets.", "dim"))
@@ -1817,6 +1824,7 @@ def cmd_init(
             workspace,
             prompt=_prompt,
             echo=_wizard_echo,
+            defaults=defaults,
             on_answer=_on_answer,
             reconfigure=reconfigure,
             force_prompt_fields=force_prompt_fields,
@@ -2121,8 +2129,13 @@ def cmd_eval_init(
     if _maybe_explain_leaf(("eval", "init"), explain):
         return
 
+    from agentops.core.config_loader import load_agentops_config
     from agentops.pipeline.azd_runner import AzdBackendError
-    from agentops.services.azd_eval_init import run_azd_eval_init
+    from agentops.services.azd_eval_init import (
+        ensure_local_evaluator_model_env,
+        recommend_evaluators_for_config,
+        run_azd_eval_init,
+    )
 
     workspace = directory.resolve()
     config_path = _resolve_eval_config_path(config)
@@ -2130,8 +2143,48 @@ def cmd_eval_init(
         config_path = workspace / config_path
 
     try:
+        loaded_config = load_agentops_config(config_path)
+        target = loaded_config.resolved_target()
+        if target.kind not in {"foundry_prompt", "foundry_hosted"}:
+            selection = recommend_evaluators_for_config(
+                config_path=config_path,
+                dataset=dataset,
+            )
+            typer.echo(
+                f"{_cli_label('AgentOps eval init')}: local HTTP/model target detected; "
+                "azd eval assets are not required."
+            )
+            typer.echo(f"{_cli_label('Evaluator recommendation')}: {selection.source}")
+            for signal in selection.signals:
+                typer.echo(f" {style('-', 'dim')} {signal}")
+            if selection.names:
+                typer.echo(f"{_cli_label('Evaluators')}: {', '.join(selection.names)}")
+            model_env = ensure_local_evaluator_model_env(
+                workspace=workspace,
+                selection=selection,
+            )
+            if model_env.configured:
+                action = "configured" if model_env.changed_keys else "using"
+                typer.echo(
+                    f"{_cli_label('Evaluator model')}: {action} "
+                    f"{model_env.deployment} ({model_env.model})"
+                )
+                if model_env.changed_keys and model_env.env_path is not None:
+                    typer.echo(
+                        f" {style('-', 'dim')} saved "
+                        f"{', '.join(model_env.changed_keys)} to "
+                        f"{_cli_path(model_env.env_path)}"
+                    )
+            elif selection.names and model_env.source != "not needed":
+                typer.echo(
+                    f"{_cli_warn('Warning')}: could not auto-discover an evaluator "
+                    "model deployment. Set AZURE_OPENAI_DEPLOYMENT and "
+                    "AZURE_OPENAI_MODEL_NAME before `agentops eval run`."
+                )
+            typer.echo(f"{_cli_label('Next')}: {_cli_command('agentops eval run')}")
+            return
         typer.echo(
-            f"{_cli_label('azd eval init')}: checking/generating eval.yaml "
+            f"{_cli_label('azd eval generate')}: checking/generating eval.yaml "
             "(this can take a few minutes on the first run)"
         )
         result = run_azd_eval_init(
@@ -2148,9 +2201,9 @@ def cmd_eval_init(
         raise typer.Exit(code=1) from exc
 
     if result.command_ran:
-        typer.echo(f"{_cli_label('azd eval init')}: completed")
+        typer.echo(f"{_cli_label('azd eval generate')}: completed")
     else:
-        typer.echo(f"{_cli_label('azd eval init')}: existing recipe reused")
+        typer.echo(f"{_cli_label('azd eval generate')}: existing recipe reused")
     if result.evaluators:
         typer.echo(f"{_cli_label('Evaluator recommendation')}: {result.evaluator_source}")
         for signal in result.evaluator_signals:
@@ -2346,6 +2399,17 @@ def cmd_assert_run(
             ),
         ),
     ] = False,
+    cached: Annotated[
+        bool,
+        typer.Option(
+            "--cached",
+            help=(
+                "Reuse cached inference/judge rows from a previous run with the "
+                "same run id. By default ASSERT re-runs inference against the live "
+                "target each time so the gate always exercises the current agent."
+            ),
+        ),
+    ] = False,
     explain: Annotated[str | None, typer.Argument(hidden=True)] = None,
 ) -> None:
     """Invoke the ASSERT (assert-ai) CLI and normalize its results."""
@@ -2403,6 +2467,7 @@ def cmd_assert_run(
     resolved_suite: str | None = suite
     resolved_run_id: str | None = run_id
     fail_on_violations = True
+    subprocess_env: dict[str, str] | None = None
 
     if cfg.assert_run is not None:
         if eval_config_path is None:
@@ -2414,6 +2479,7 @@ def cmd_assert_run(
         if resolved_run_id is None:
             resolved_run_id = cfg.assert_run.run_id
         fail_on_violations = cfg.assert_run.fail_on_violations
+        subprocess_env = dict(cfg.assert_run.env)
     if no_gate:
         fail_on_violations = False
 
@@ -2428,6 +2494,12 @@ def cmd_assert_run(
         typer.echo(
             f"  suite={resolved_suite or '<auto>'} run_id={resolved_run_id or '<auto>'}"
         )
+    if cached:
+        typer.echo("  cache: reusing prior inference/judge rows when available")
+    else:
+        typer.echo("  cache: forcing fresh inference against the live target")
+
+    assert_extra_args = None if cached else ["--force-stage", "inference"]
 
     try:
         result = run_assert(
@@ -2436,6 +2508,8 @@ def cmd_assert_run(
             results_dir=resolved_results_dir,
             suite=resolved_suite,
             run_id=resolved_run_id,
+            env=subprocess_env,
+            extra_args=assert_extra_args,
         )
     except AssertRunnerError as exc:
         typer.echo(f"{_cli_error('Error')}: {exc}", err=True)
@@ -2471,9 +2545,15 @@ def cmd_assert_run(
             violations = bucket.get("violations", 0)
             total = bucket.get("total", 0)
             skipped = bucket.get("skipped", 0)
-            marker = _cli_ok("OK") if violations == 0 else _cli_error("VIOLATIONS")
             suffix = f" (skipped={skipped})" if skipped else ""
-            typer.echo(f"  {name}: {violations}/{total}{suffix} {marker}")
+            if violations == 0:
+                clean = max(total - skipped, 0)
+                typer.echo(f"  {name}: {clean}/{total} clean{suffix} {_cli_ok('OK')}")
+            else:
+                typer.echo(
+                    f"  {name}: {violations}/{total} violating{suffix} "
+                    f"{_cli_error('VIOLATIONS')}"
+                )
 
     typer.echo("")
     typer.echo(_cli_heading("Inspect details"))
@@ -2666,6 +2746,7 @@ def cmd_redteam_run(
                 err=True,
             )
             raise typer.Exit(code=1)
+    _apply_http_redteam_defaults(resolved_target, cfg)
 
     if output_path is not None and not output_path.is_absolute():
         output_path = (workspace / output_path).resolve()
@@ -2786,6 +2867,21 @@ def _derive_redteam_target_from_agent(agent: str | None) -> dict[str, Any]:
     return {"agent": agent}
 
 
+def _apply_http_redteam_defaults(target: dict[str, Any], cfg: AgentOpsConfig) -> None:
+    if "endpoint" not in target:
+        return
+    if cfg.request_field:
+        target.setdefault("request_field", cfg.request_field)
+    if cfg.response_field:
+        target.setdefault("response_field", cfg.response_field)
+    if cfg.response_mode:
+        target.setdefault("response_mode", cfg.response_mode)
+    if cfg.headers:
+        target.setdefault("headers", cfg.headers)
+    if cfg.stream:
+        target.setdefault("stream", cfg.stream.model_dump(exclude_none=True))
+
+
 def _run_flat_schema_eval(
     *,
     config_path: Path,

{agentops_accelerator-0.4.4 → agentops_accelerator-0.4.5}/src/agentops/core/agentops_config.py

@@ -462,6 +462,14 @@ class AssertRunConfig(BaseModel):
             "results without gating the pipeline."
         ),
     )
+    env: Dict[str, str] = Field(
+        default_factory=dict,
+        description=(
+            "Optional non-secret environment variables passed only to the "
+            "assert-ai subprocess, for example AZURE_API_BASE or "
+            "AZURE_API_VERSION."
+        ),
+    )
 
     model_config = ConfigDict(extra="forbid")
 

{agentops_accelerator-0.4.4 → agentops_accelerator-0.4.5}/src/agentops/pipeline/publisher.py

@@ -110,12 +110,18 @@ def _build_instance_rows(result: RunResult) -> List[Dict[str, Any]]:
     for row in result.rows:
         payload: Dict[str, Any] = {
             "line_number": row.row_index,
-            "input": row.input,
-            "response": row.response,
-            "ground_truth": row.expected or "",
+            "inputs.input": row.input,
+            "inputs.response": row.response,
+            "inputs.ground_truth": row.expected or "",
         }
         for metric in row.metrics:
             if metric.value is not None:
-                payload[metric.name] = metric.value
+                payload[f"outputs.{metric.name}.score"] = metric.value
+                if not metric.name.endswith("_latency_seconds"):
+                    payload[f"metric.{metric.name}"] = metric.value
+            if metric.reason:
+                payload[f"outputs.{metric.name}.reason"] = metric.reason
+            if metric.error:
+                payload[f"outputs.{metric.name}.error"] = metric.error
         rows.append(payload)
     return rows

{agentops_accelerator-0.4.4 → agentops_accelerator-0.4.5}/src/agentops/pipeline/runtime.py

@@ -67,6 +67,10 @@ def _credential() -> Any:
 _REASONING_MODEL_PREFIXES = ("gpt-5", "o1", "o3", "o4")
 
 
+def _evaluator_model_name() -> Optional[str]:
+    return os.getenv("AZURE_OPENAI_MODEL_NAME") or os.getenv("AZURE_AI_MODEL_NAME")
+
+
 def _model_config() -> Dict[str, Any]:
     from agentops.utils.azure_endpoints import (
         derive_openai_endpoint_from_project,
@@ -166,7 +170,9 @@ def load_evaluator(preset: EvaluatorPreset) -> EvaluatorRuntime:
     if preset.class_name in _AI_ASSISTED:
         model_config = _model_config()
         init_kwargs["model_config"] = model_config
-        if _is_reasoning_model_deployment(model_config.get("azure_deployment")):
+        if _is_reasoning_model_deployment(
+            _evaluator_model_name() or model_config.get("azure_deployment")
+        ):
             init_kwargs["is_reasoning_model"] = True
     if preset.class_name in _SAFETY:
         init_kwargs["azure_ai_project"] = _project_endpoint()