azure-ai-evaluation 1.13.4__tar.gz → 1.13.5__tar.gz

{azure_ai_evaluation-1.13.4 → azure_ai_evaluation-1.13.5}/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Release History
 
+## 1.13.5 (2025-11-10)
+
+### Bugs Fixed
+
+- **TaskAdherenceEvaluator:** treat tool definitions as optional so evaluations with only query/response inputs no longer raise “Either 'conversation' or individual inputs must be provided.”
+
 ## 1.13.4 (2025-11-10)
 
 ### Bugs Fixed

{azure_ai_evaluation-1.13.4/azure_ai_evaluation.egg-info → azure_ai_evaluation-1.13.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: azure-ai-evaluation
-Version: 1.13.4
+Version: 1.13.5
 Summary: Microsoft Azure Evaluation Library for Python
 Home-page: https://github.com/Azure/azure-sdk-for-python
 Author: Microsoft Corporation
@@ -418,6 +418,12 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 
 # Release History
 
+## 1.13.5 (2025-11-10)
+
+### Bugs Fixed
+
+- **TaskAdherenceEvaluator:** treat tool definitions as optional so evaluations with only query/response inputs no longer raise “Either 'conversation' or individual inputs must be provided.”
+
 ## 1.13.4 (2025-11-10)
 
 ### Bugs Fixed

azure_ai_evaluation-1.13.5/TROUBLESHOOTING.md

@@ -0,0 +1,130 @@
+# Troubleshoot AI Evaluation SDK Issues
+
+This guide walks you through how to investigate failures, common errors in the `azure-ai-evaluation` SDK, and steps to mitigate these issues.
+
+## Table of Contents
+
+- [Handle Evaluate API Errors](#handle-evaluate-api-errors)
+  - [Troubleshoot Remote Tracking Issues](#troubleshoot-remote-tracking-issues)
+  - [Troubleshoot Column Mapping Issues](#troubleshoot-column-mapping-issues)
+  - [Troubleshoot Safety Evaluator Issues](#troubleshoot-safety-evaluator-issues)
+  - [Troubleshoot Quality Evaluator Issues](#troubleshoot-quality-evaluator-issues)
+- [Handle Simulation Errors](#handle-simulation-errors)
+  - [Adversarial Simulation Supported Regions](#adversarial-simulation-supported-regions)
+  - [Need to generate simulations for specific harm type](#need-to-generate-simulations-for-specific-harm-type)
+  - [Simulator is slow](#simulator-is-slow)
+- [Handle RedTeam Errors](#handle-redteam-errors)
+  - [Permission or authentication failures](#permission-or-authentication-failures)
+  - [Target resource not found](#target-resource-not-found)
+  - [Agent name not found](#agent-name-not-found)
+  - [Insufficient Storage Permissions](#insufficient-storage-permissions)
+  - [PyRIT "Error sending prompt" message](#pyrit-error-sending-prompt-message)
+- [Logging](#logging)
+- [Get Additional Help](#get-additional-help)
+
+## Handle Evaluate API Errors
+
+### Troubleshoot Remote Tracking Issues
+
+- Before running `evaluate()`, to ensure that you can enable logging and tracing to your Azure AI project, make sure you are first logged in by running `az login`.
+
+- Ensure that you assign the proper permissions to the storage account linked to your Azure AI Studio hub. This can be done with the following command. More information can be found [here](https://aka.ms/credentialleshub).
+
+    ```Shell
+    # <mySubscriptionID>: Subscription ID of the Azure AI Studio hub's linked storage account (available in Azure AI hub resource view in Azure Portal).
+    # <myResourceGroupName>: Resource group of the Azure AI Studio hub's linked storage account.
+    # <user-id>: User object ID for role assignment (retrieve with "az ad user show" command).
+
+    az role assignment create --role "Storage Blob Data Contributor" --scope /subscriptions/<mySubscriptionID>/resourceGroups/<myResourceGroupName> --assignee-principal-type User --assignee-object-id "<user-id>"
+    ```
+
+- Additionally, if you're using a virtual network or private link, and your evaluation run upload fails because of that, check out this [guide](https://docs.microsoft.com/azure/machine-learning/how-to-enable-studio-virtual-network#access-data-using-the-studio).
+
+### Troubleshoot Column Mapping Issues
+
+- When using `column_mapping` parameter in evaluators, ensure all keys and values are non-empty strings and contain only alphanumeric characters. Empty strings, non-string values, or non-alphanumeric characters can cause serialization errors and issues in downstream applications. Example of valid mapping: `{"query": "${data.query}", "response": "${data.response}"}`.
+
+### Troubleshoot Safety Evaluator Issues
+
+- Risk and safety evaluators depend on the Azure AI Studio safety evaluation backend service. For a list of supported regions, please refer to the documentation [here](https://aka.ms/azureaisafetyeval-regionsupport).
+- If you encounter a 403 Unauthorized error when using safety evaluators, verify that you have the `Contributor` role assigned to your Azure AI project. `Contributor` role is currently required to run safety evaluations.
+
+## Handle Simulation Errors
+
+### Adversarial Simulation Supported Regions
+
+Adversarial simulators use Azure AI Studio safety evaluation backend service to generate an adversarial dataset against your application. For a list of supported regions, please refer to the documentation [here](https://aka.ms/azureaiadvsimulator-regionsupport).
+
+### Need to generate simulations for specific harm type
+
+The Adversarial simulator does not support selecting individual harms, instead we recommend running the `AdversarialSimulator` for 4x the number of specific harms as the `max_simulation_results`
+
+### Simulator is slow
+
+Identify the type of simulations being run (adversarial or non-adversarial).
+Adjust parameters such as `api_call_retry_sleep_sec`, `api_call_delay_sec`, and `concurrent_async_task`. Please note that rate limits to llm calls can be both tokens per minute and requests per minute.
+
+## Handle RedTeam errors
+
+### Permission or authentication failures
+- Run `az login` in the active shell before starting the scan and ensure the account has the **Azure AI User** role plus the `Storage Blob Data Contributor` assignment on the linked storage account. Both are required to create evaluation runs and upload artifacts.
+- In secured hubs, confirm the linked storage account allows access from your network (or private endpoint) and that Entra ID authentication is enabled on the storage resource.
+- If the helper warns `This may be due to missing environment variables or insufficient permissions.`, double-check the `AZURE_PROJECT_ENDPOINT`, `AGENT_NAME`, and storage role assignments before retrying.
+
+### Target resource not found
+- When initializing an Azure OpenAI deployment directly as the `target`, specify `azure_endpoint` as `https://<hub>.openai.azure.com/openai/deployments/<deployment_name>/chat/completions?api-version=2025-01-01-preview`.
+- If you instantiate `AzureOpenAI`, use the resource-level endpoint format `https://<hub>.openai.azure.com/` and ensure the deployment name plus API version match an active deployment.
+- A cloud run error such as `Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}` when creating the eval group can also indicate that `azure-ai-projects>=2.0.0b1` is not installed. Upgrade to that version or later to access the preview APIs used by Red Team.
+
+### Agent name not found
+- `(not_found) Agent <name> doesn’t exist` means the Azure AI project could not resolve the agent `name`. Names are case sensitive and differ from display names.
+- Verify the `AZURE_PROJECT_ENDPOINT` points to the correct project and that the agent is published there.
+- Requires `DefaultAzureCredential` from `azure.identity` and `AIProjectClient` from `azure.ai.projects`.
+- Use the following helper to list agents in the current project and confirm the `name` column matches your `AGENT_NAME` value:
+
+    ```python
+    def list_project_agents(endpoint: str | None = None) -> None:
+        project_endpoint = endpoint or os.environ.get("AZURE_PROJECT_ENDPOINT") or ""
+        if not project_endpoint:
+            print("Set AZURE_PROJECT_ENDPOINT before listing agents.")
+            return
+        with DefaultAzureCredential() as project_credential:
+            with AIProjectClient(
+                endpoint=project_endpoint,
+                credential=project_credential,
+                api_version="2025-11-15-preview",
+            ) as project_client:
+                agents = list(project_client.agents.list())
+        if not agents:
+            print(f"No agents found in project: {project_endpoint}")
+            return
+        print(f"Agents in {project_endpoint}:")
+        for agent in agents:
+            display_name = agent.get("display_name") if isinstance(agent, dict) else getattr(agent, "display_name", "")
+            name = agent.get("name") if isinstance(agent, dict) else getattr(agent, "name", "")
+            print(f"- name: {name} | display_name: {display_name}")
+    ```
+
+### Insufficient Storage Permissions
+- `WARNING: Failed to log artifacts to MLFlow: (UserError) Failed to upload evaluation run to the cloud due to insufficient permission to access the storage` means the linked storage account is missing the necessary assignments.
+- Portal steps:
+  1. Open the resource group tied to the Azure AI Project in the Azure Portal.
+  2. Locate the linked storage account(s).
+  3. Select each storage account and choose **Access control (IAM)**.
+  4. Grant the affected identity the **Storage Blob Data Contributor** role.
+- Prefer CLI? Reuse the `az role assignment create` command described in [Troubleshoot Remote Tracking Issues](#troubleshoot-remote-tracking-issues).
+
+### PyRIT "Error sending prompt" message
+- `Exception: Error sending prompt with conversation ID: <guid>` is raised by PyRIT when a target LLM call fails inside the `PromptSendingOrchestrator`. The runner retries the conversation up to the configured limit, so occasional occurrences usually resolve automatically.
+- Common triggers include transient network issues, 429 throttling, or 5xx responses from the target deployment. Even if retries succeed you will still see the stack trace in notebook output.
+- Inspect the `redteam.log` file written to the scan output directory (typically `<working dir>/runs/<scan_id>/redteam.log`) for the underlying exception and HTTP status. Increase verbosity with `DEBUG=True` for deeper diagnostics.
+- Running in Azure AI Studio? Navigate to **Evaluate > Red Team > <run name> > Logs**, download `redteam.log`, and search for the conversation ID to inspect the payload.
+- If one conversation ID keeps failing after retries, verify the target credentials, check deployment health, and review Azure OpenAI quota or rate-limit alerts in the Azure portal.
+
+## Logging
+
+You can set logging level via environment variable `PF_LOGGING_LEVEL`, valid values include `CRITICAL`, `ERROR`, `WARNING`, `INFO`, `DEBUG`; default is `INFO`.
+
+## Get Additional Help
+
+Additional information on ways to reach out for support can be found in the [SUPPORT.md](https://github.com/Azure/azure-sdk-for-python/blob/main/SUPPORT.md) at the root of the repo.

{azure_ai_evaluation-1.13.4 → azure_ai_evaluation-1.13.5}/azure/ai/evaluation/_evaluators/_common/_base_eval.py

@@ -3,6 +3,7 @@
 # ---------------------------------------------------------
 
 import inspect
+import logging
 from abc import ABC, abstractmethod
 import json
 import copy
@@ -46,6 +47,8 @@ P = ParamSpec("P")
 T = TypeVar("T")
 T_EvalValue = TypeVar("T_EvalValue")
 
+logger = logging.getLogger(__name__)
+
 
 class DerivedEvalInput(TypedDict, total=False):
     """The eval input generated by EvaluatorBase._derive_conversation_starter."""
@@ -593,7 +596,7 @@ class EvaluatorBase(ABC, Generic[T_EvalValue]):
         try:
             eval_input_list = self._convert_kwargs_to_eval_input(**kwargs)
         except Exception as e:
-            print(f"Error converting kwargs to eval_input_list: {e}")
+            logger.error(f"Error converting kwargs to eval_input_list: {e}")
             raise e
         per_turn_results = []
         # Evaluate all inputs.
@@ -630,7 +633,7 @@ class EvaluatorBase(ABC, Generic[T_EvalValue]):
                             else:
                                 result[result_key] = EVALUATION_PASS_FAIL_MAPPING[False]
             except Exception as e:
-                print(f"Error calculating binary result: {e}")
+                logger.warning(f"Error calculating binary result: {e}")
             per_turn_results.append(result)
         # Return results as-is if only one result was produced.
 

{azure_ai_evaluation-1.13.4 → azure_ai_evaluation-1.13.5}/azure/ai/evaluation/_evaluators/_task_adherence/_task_adherence.py

@@ -63,7 +63,7 @@ class TaskAdherenceEvaluator(PromptyEvaluatorBase[Union[str, float]]):
 
     _PROMPTY_FILE = "task_adherence.prompty"
     _RESULT_KEY = "task_adherence"
-    _OPTIONAL_PARAMS = []
+    _OPTIONAL_PARAMS = ["tool_definitions"]
 
     _DEFAULT_TASK_ADHERENCE_SCORE = 0
 
@@ -175,7 +175,7 @@ class TaskAdherenceEvaluator(PromptyEvaluatorBase[Union[str, float]]):
                         if isinstance(content, list):
                             for item in content:
                                 if isinstance(item, dict):
-                                    if item.get("type") == "text":
+                                    if item.get("type", None) in ("text", "input_text", "output_text"):
                                         assistant_parts.append(item.get("text", ""))
                                     elif item.get("type") == "tool_call":
                                         tool_parts.append(str(item.get("tool_call", "")))

{azure_ai_evaluation-1.13.4 → azure_ai_evaluation-1.13.5}/azure/ai/evaluation/_version.py

@@ -3,4 +3,4 @@
 # ---------------------------------------------------------
 # represents upcoming version
 
-VERSION = "1.13.4"
+VERSION = "1.13.5"

{azure_ai_evaluation-1.13.4 → azure_ai_evaluation-1.13.5/azure_ai_evaluation.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: azure-ai-evaluation
-Version: 1.13.4
+Version: 1.13.5
 Summary: Microsoft Azure Evaluation Library for Python
 Home-page: https://github.com/Azure/azure-sdk-for-python
 Author: Microsoft Corporation
@@ -418,6 +418,12 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 
 # Release History
 
+## 1.13.5 (2025-11-10)
+
+### Bugs Fixed
+
+- **TaskAdherenceEvaluator:** treat tool definitions as optional so evaluations with only query/response inputs no longer raise “Either 'conversation' or individual inputs must be provided.”
+
 ## 1.13.4 (2025-11-10)
 
 ### Bugs Fixed

{azure_ai_evaluation-1.13.4 → azure_ai_evaluation-1.13.5}/azure_ai_evaluation.egg-info/SOURCES.txt

@@ -327,13 +327,22 @@ samples/red_team_samples.py
 samples/red_team_skip_upload.py
 samples/semantic_kernel_red_team_agent_sample.py
 samples/agent_evaluators/agent_evaluation.ipynb
+samples/agent_evaluators/coherence.ipynb
+samples/agent_evaluators/fluency.ipynb
+samples/agent_evaluators/groundedness.ipynb
 samples/agent_evaluators/instructions.md
 samples/agent_evaluators/intent_resolution.ipynb
+samples/agent_evaluators/relevance.ipynb
 samples/agent_evaluators/response_completeness.ipynb
 samples/agent_evaluators/sample_synthetic_conversations.jsonl
 samples/agent_evaluators/task_adherence.ipynb
+samples/agent_evaluators/task_completion.ipynb
 samples/agent_evaluators/task_navigation_efficiency.ipynb
 samples/agent_evaluators/tool_call_accuracy.ipynb
+samples/agent_evaluators/tool_input_accuracy.ipynb
+samples/agent_evaluators/tool_output_utilization.ipynb
+samples/agent_evaluators/tool_selection.ipynb
+samples/agent_evaluators/tool_success.ipynb
 samples/agent_evaluators/user_functions.py
 samples/data/custom_objectives_with_context_example.json
 samples/data/evaluate_test_data.jsonl

azure_ai_evaluation-1.13.5/samples/agent_evaluators/coherence.ipynb

@@ -0,0 +1,204 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "da22a6e7",
+   "metadata": {},
+   "source": [
+    "# Coherence Evaluator\n",
+    "\n",
+    "### Getting Started\n",
+    "This sample demonstrates how to use the Coherence evaluator to assess the quality and readability of AI-generated responses. The evaluator measures the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language.\n",
+    "\n",
+    "Before you begin:\n",
+    "```bash\n",
+    "pip install azure-ai-evaluation\n",
+    "```\n",
+    "Set these environment variables with your own values:\n",
+    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
+    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
+    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
+    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "48f8cb64",
+   "metadata": {},
+   "source": [
+    "The Coherence evaluator assesses the ability of the language model to generate text that reads naturally, flows smoothly, and resembles human-like language in its responses. It measures the readability and user-friendliness of the model's generated responses.\n",
+    "\n",
+    "Coherence scores range from 1 to 5:\n",
+    "\n",
+    "<pre>\n",
+    "Score 1: Very Poor - The response is completely incoherent with no logical flow or connection between ideas.\n",
+    "Score 2: Poor - The response has significant coherence issues with disjointed ideas and unclear transitions.\n",
+    "Score 3: Fair - The response is somewhat coherent but has noticeable gaps in logical flow or organization.\n",
+    "Score 4: Good - The response is mostly coherent with good flow and logical connections between ideas.\n",
+    "Score 5: Excellent - The response is perfectly coherent with smooth, natural flow and clear logical progression.\n",
+    "</pre>\n",
+    "\n",
+    "The evaluation requires one of the following input patterns:\n",
+    "\n",
+    "**Pattern 1: Query-Response Evaluation**\n",
+    "- Query: The user's question or prompt. (string)\n",
+    "- Response: The AI-generated response to be evaluated for coherence. (string)\n",
+    "\n",
+    "**Pattern 2: Conversation Evaluation**\n",
+    "- Conversation: A multi-turn conversation where each turn is evaluated for coherence. (dict with \"messages\" key)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "542c1340",
+   "metadata": {},
+   "source": [
+    "### Initialize Coherence Evaluator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "35799119",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from azure.ai.evaluation import CoherenceEvaluator, AzureOpenAIModelConfiguration\n",
+    "from pprint import pprint\n",
+    "\n",
+    "model_config = AzureOpenAIModelConfiguration(\n",
+    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
+    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
+    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
+    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
+    ")\n",
+    "\n",
+    "coherence_evaluator = CoherenceEvaluator(model_config=model_config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b10aff00",
+   "metadata": {},
+   "source": [
+    "### Samples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "724adfc8",
+   "metadata": {},
+   "source": [
+    "#### Query and Response as Strings (str)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "82a1be93",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"Can you explain how machine learning works?\"\n",
+    "response = \"Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed. It works by analyzing data patterns to make predictions or decisions. First, the system is trained on a dataset, which allows it to identify patterns and relationships. Then, it uses these learned patterns to make predictions on new, unseen data. The accuracy improves over time as the model processes more data and refines its understanding.\"\n",
+    "\n",
+    "result = coherence_evaluator(query=query, response=response)\n",
+    "pprint(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bc250127",
+   "metadata": {},
+   "source": [
+    "#### Example of Poor Coherence"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62ad5478",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Poor coherence example\n",
+    "query = \"What is blockchain technology?\"\n",
+    "response = \"Blockchain is decentralized. Bitcoin uses it. Blocks contain data. Cryptography is involved. Banks might use blockchain. It's secure because of the chain. Mining is how new blocks are added. Digital currencies exist. The ledger is distributed.\"\n",
+    "\n",
+    "result = coherence_evaluator(query=query, response=response)\n",
+    "pprint(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1d22e3d6",
+   "metadata": {},
+   "source": [
+    "#### Conversation as Dict with Messages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "556c197b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Multi-turn conversation\n",
+    "conversation = {\n",
+    "    \"messages\": [\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"I'm planning a trip to Japan. What should I know?\"\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Japan is a fascinating destination with a rich cultural heritage. First, you should know that the best times to visit are spring (March-May) for cherry blossoms and fall (September-November) for autumn colors. The country has an excellent public transportation system, particularly the Shinkansen bullet train network. Additionally, learning a few basic Japanese phrases will be appreciated by locals, though many signs in major cities have English translations.\"\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"What about the food?\"\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Japanese cuisine is incredibly diverse and delicious. Beyond the well-known sushi and ramen, you should try regional specialties like okonomiyaki in Osaka or motsunabe in Fukuoka. Most restaurants display plastic food models in their windows, which makes ordering easier even with a language barrier. It's also important to note that tipping is not customary in Japan and can sometimes be considered rude.\"\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Are there any cultural customs I should be aware of?\"\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Yes, there are several important customs to observe. Always remove your shoes when entering homes, temples, and some traditional restaurants. Bowing is a common greeting, though a slight nod is acceptable for foreigners. When using chopsticks, never stick them upright in rice as this resembles a funeral ritual. Finally, be mindful of noise levels on public transportation, as talking on phones or having loud conversations is considered impolite.\"\n",
+    "        }\n",
+    "    ]\n",
+    "}\n",
+    "\n",
+    "result = coherence_evaluator(conversation=conversation)\n",
+    "pprint(result)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "test_agent_evaluator_prp",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}