azure-ai-evaluation 1.0.0b4__py3-none-any.whl → 1.0.1__py3-none-any.whl

azure/ai/evaluation/_evaluators/_groundedness/groundedness_without_query.prompty

@@ -0,0 +1,99 @@
+---
+name: Groundedness
+description: Evaluates groundedness score for RAG scenario
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 800
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: text
+
+inputs:
+  response:
+    type: string
+  context:
+    type: string
+
+---
+system:
+# Instruction
+## Goal
+### You are an expert in evaluating the quality of a RESPONSE from an intelligent system based on provided definition and data. Your goal will involve answering the questions below using the information provided.
+- **Definition**: You are given a definition of the communication trait that is being evaluated to help guide your Score.
+- **Data**: Your input data include CONTEXT and RESPONSE.
+- **Tasks**: To complete your evaluation you will be asked to evaluate the Data in different ways.
+
+user:
+# Definition
+**Groundedness** refers to how faithfully a response adheres to the information provided in the CONTEXT, ensuring that all content is directly supported by the context without introducing unsupported information or omitting critical details. It evaluates the fidelity and precision of the response in relation to the source material.
+
+# Ratings
+## [Groundedness: 1] (Completely Ungrounded Response)
+**Definition:** The response is entirely unrelated to the CONTEXT, introducing topics or information that have no connection to the provided material.
+
+**Examples:**
+  **Context:** The company's profits increased by 20% in the last quarter.
+  **Response:** I enjoy playing soccer on weekends with my friends.
+
+  **Context:** The new smartphone model features a larger display and improved battery life.
+  **Response:** The history of ancient Egypt is fascinating and full of mysteries.
+
+## [Groundedness: 2] (Contradictory Response)
+**Definition:** The response directly contradicts or misrepresents the information provided in the CONTEXT.
+
+**Examples:**
+  **Context:** The company's profits increased by 20% in the last quarter.
+  **Response:** The company's profits decreased by 20% in the last quarter.
+
+  **Context:** The new smartphone model features a larger display and improved battery life.
+  **Response:** The new smartphone model has a smaller display and shorter battery life.
+
+## [Groundedness: 3] (Accurate Response with Unsupported Additions)
+**Definition:** The response accurately includes information from the CONTEXT but adds details, opinions, or explanations that are not supported by the provided material.
+
+**Examples:**
+  **Context:** The company's profits increased by 20% in the last quarter.
+  **Response:** The company's profits increased by 20% in the last quarter due to their aggressive marketing strategy.
+
+  **Context:** The new smartphone model features a larger display and improved battery life.
+  **Response:** The new smartphone model features a larger display, improved battery life, and comes with a free case.
+
+## [Groundedness: 4] (Incomplete Response Missing Critical Details)
+**Definition:** The response contains information from the CONTEXT but omits essential details that are necessary for a comprehensive understanding of the main point.
+
+**Examples:**
+  **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.      
+  **Response:** The company's profits increased by 20% in the last quarter.
+
+  **Context:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.        
+  **Response:** The new smartphone model features a larger display and improved battery life.
+
+## [Groundedness: 5] (Fully Grounded and Complete Response)
+**Definition:** The response is entirely based on the CONTEXT, accurately and thoroughly conveying all essential information without introducing unsupported details or omitting critical points.
+
+**Examples:**
+  **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.      
+  **Response:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.     
+
+  **Context:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.        
+  **Response:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.  
+
+
+# Data
+CONTEXT: {{context}}
+RESPONSE: {{response}}
+
+
+# Tasks
+## Please provide your assessment Score for the previous RESPONSE in relation to the CONTEXT based on the Definitions above. Your output should include the following information:
+- **ThoughtChain**: To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your ThoughtChain with "Let's think step by step:".
+- **Explanation**: a very short explanation of why you think the input Data should get that Score.
+- **Score**: based on your previous analysis, provide your Score. The Score you give MUST be a integer score (i.e., "1", "2"...) based on the levels of the definitions.
+
+
+## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
+# Output

azure/ai/evaluation/_evaluators/_meteor/_meteor.py

@@ -1,11 +1,10 @@
 # ---------------------------------------------------------
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # ---------------------------------------------------------
-import nltk
 from nltk.translate.meteor_score import meteor_score
 from promptflow._utils.async_utils import async_run_allowing_running_loop
 
-from azure.ai.evaluation._common.utils import nltk_tokenize
+from azure.ai.evaluation._common.utils import nltk_tokenize, ensure_nltk_data_downloaded
 
 
 class _AsyncMeteorScoreEvaluator:
@@ -14,10 +13,7 @@ class _AsyncMeteorScoreEvaluator:
         self._beta = beta
         self._gamma = gamma
 
-        try:
-            nltk.find("corpora/wordnet.zip")
-        except LookupError:
-            nltk.download("wordnet")
+        ensure_nltk_data_downloaded()
 
     async def __call__(self, *, ground_truth: str, response: str, **kwargs):
         reference_tokens = nltk_tokenize(ground_truth)
@@ -38,7 +34,7 @@ class _AsyncMeteorScoreEvaluator:
 
 class MeteorScoreEvaluator:
     """
-    Evaluator that computes the METEOR Score between two strings.
+    Calculates the METEOR score for a given response and ground truth.
 
     The METEOR (Metric for Evaluation of Translation with Explicit Ordering) score grader evaluates generated text by
     comparing it to reference texts, focusing on precision, recall, and content alignment. It addresses limitations of
@@ -46,6 +42,12 @@ class MeteorScoreEvaluator:
     word stems to more accurately capture meaning and language variations. In addition to machine translation and
     text summarization, paraphrase detection is an optimal use case for the METEOR score.
 
+    Use the METEOR score when you want a more linguistically informed evaluation metric that captures not only
+    n-gram overlap but also accounts for synonyms, stemming, and word order. This is particularly useful for evaluating
+    tasks like machine translation, text summarization, and text generation.
+
+    The METEOR score ranges from 0 to 1, with 1 indicating a perfect match.
+
     :param alpha: The METEOR score alpha parameter. Default is 0.9.
     :type alpha: float
     :param beta: The METEOR score beta parameter. Default is 3.0.
@@ -53,28 +55,19 @@ class MeteorScoreEvaluator:
     :param gamma: The METEOR score gamma parameter. Default is 0.5.
     :type gamma: float
 
-    **Usage**
-
-    .. code-block:: python
-
-        eval_fn = MeteorScoreEvaluator(
-            alpha=0.9,
-            beta=3.0,
-            gamma=0.5
-        )
-        result = eval_fn(
-            response="Tokyo is the capital of Japan.",
-            ground_truth="The capital of Japan is Tokyo.")
+    .. admonition:: Example:
 
-    **Output format**
-
-    .. code-block:: python
-
-        {
-            "meteor_score": 0.62
-        }
+        .. literalinclude:: ../samples/evaluation_samples_evaluate.py
+            :start-after: [START meteor_score_evaluator]
+            :end-before: [END meteor_score_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call a MeteorScoreEvaluator with alpha of 0.8.
     """
 
+    id = "azureml://registries/azureml/models/Meteor-Score-Evaluator/versions/3"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+
     def __init__(self, alpha: float = 0.9, beta: float = 3.0, gamma: float = 0.5):
         self._async_evaluator = _AsyncMeteorScoreEvaluator(alpha=alpha, beta=beta, gamma=gamma)
 
@@ -87,7 +80,7 @@ class MeteorScoreEvaluator:
         :keyword ground_truth: The ground truth to be compared against.
         :paramtype ground_truth: str
         :return: The METEOR score.
-        :rtype: dict
+        :rtype: Dict[str, float]
         """
         return async_run_allowing_running_loop(
             self._async_evaluator, ground_truth=ground_truth, response=response, **kwargs

azure/ai/evaluation/_evaluators/_multimodal/__init__.py

@@ -0,0 +1,20 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+from ._content_safety_multimodal import ContentSafetyMultimodalEvaluator
+from ._content_safety_multimodal_base import ContentSafetyMultimodalEvaluatorBase
+from ._hate_unfairness import HateUnfairnessMultimodalEvaluator
+from ._self_harm import SelfHarmMultimodalEvaluator
+from ._sexual import SexualMultimodalEvaluator
+from ._violence import ViolenceMultimodalEvaluator
+from ._protected_material import ProtectedMaterialMultimodalEvaluator
+
+__all__ = [
+    "ContentSafetyMultimodalEvaluator",
+    "ContentSafetyMultimodalEvaluatorBase",
+    "ViolenceMultimodalEvaluator",
+    "SexualMultimodalEvaluator",
+    "SelfHarmMultimodalEvaluator",
+    "HateUnfairnessMultimodalEvaluator",
+    "ProtectedMaterialMultimodalEvaluator",
+]

azure/ai/evaluation/_evaluators/_multimodal/_content_safety_multimodal.py

@@ -0,0 +1,132 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+import logging
+from concurrent.futures import as_completed
+from typing import Callable, Dict, List, Union
+
+from promptflow.tracing import ThreadPoolExecutorWithContext as ThreadPoolExecutor
+from azure.ai.evaluation._common._experimental import experimental
+from azure.ai.evaluation._common.utils import validate_conversation
+from ._hate_unfairness import HateUnfairnessMultimodalEvaluator
+from ._self_harm import SelfHarmMultimodalEvaluator
+from ._sexual import SexualMultimodalEvaluator
+from ._violence import ViolenceMultimodalEvaluator
+
+
+logger = logging.getLogger(__name__)
+
+
+@experimental
+class ContentSafetyMultimodalEvaluator:
+    """
+    Initialize a content safety multimodal evaluator configured to evaluate content safety metrics
+    in a multimodal scenario.
+
+    :param credential: The credential required for connecting to the Azure AI project.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :param azure_ai_project: The scope of the Azure AI project, containing the subscription ID,
+        resource group, and project name.
+    :type azure_ai_project: ~azure.ai.evaluation.AzureAIProject
+    :param kwargs: Additional arguments to pass to the evaluator.
+    :type kwargs: Any
+
+    :return: A function that evaluates multimodal chat messages and generates content safety metrics.
+
+    **Usage Example**
+
+    .. code-block:: python
+
+        azure_ai_project = {
+            "subscription_id": "<subscription_id>",
+            "resource_group_name": "<resource_group_name>",
+            "project_name": "<project_name>",
+        }
+        eval_fn = ContentSafetyMultimodalEvaluator(azure_ai_project)
+        result = eval_fn(conversation=
+            {
+                "messages": [
+                    {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "text",
+                                "text": "What's in this image?"
+                            },
+                            {
+                                "type": "image_url",
+                                "image_url": {
+                                    "url": "<image url or base64 encoded image>"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "role": "assistant",
+                        "content": "This picture shows an astronaut standing in the desert."
+                    }
+                ]
+            }
+        )
+
+    **Output Format**
+
+    .. code-block:: json
+
+        {
+            "violence": "Medium",
+            "violence_score": 5.0,
+            "violence_reason": "Some reason",
+            "sexual": "Medium",
+            "sexual_score": 5.0,
+            "sexual_reason": "Some reason",
+            "self_harm": "Medium",
+            "self_harm_score": 5.0,
+            "self_harm_reason": "Some reason",
+            "hate_unfairness": "Medium",
+            "hate_unfairness_score": 5.0,
+            "hate_unfairness_reason": "Some reason"
+        }
+
+    """
+
+    id = "content_safety_multimodal"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+
+    def __init__(self, credential, azure_ai_project, **kwargs):
+        self._parallel = kwargs.pop("_parallel", False)
+        self._evaluators: List[Callable[..., Dict[str, Union[str, float]]]] = [
+            ViolenceMultimodalEvaluator(credential=credential, azure_ai_project=azure_ai_project),
+            SexualMultimodalEvaluator(credential=credential, azure_ai_project=azure_ai_project),
+            SelfHarmMultimodalEvaluator(credential=credential, azure_ai_project=azure_ai_project),
+            HateUnfairnessMultimodalEvaluator(credential=credential, azure_ai_project=azure_ai_project),
+        ]
+
+    def __call__(self, *, conversation, **kwargs):
+        """
+        Evaluates content-safety metrics for list of messages.
+
+        :keyword conversation: The conversation contains list of messages to be evaluated.
+            Each message should have "role" and "content" keys. It supports single turn only.
+        :paramtype conversation: ~azure.ai.evaluation.Conversation
+        :return: The evaluation score based on the Content Safety Metrics.
+        :rtype: Dict[str, Union[float, str]]
+        """
+        # validate inputs
+        validate_conversation(conversation)
+        results: Dict[str, Union[str, float]] = {}
+        if self._parallel:
+            with ThreadPoolExecutor() as executor:
+                futures = {
+                    executor.submit(evaluator, conversation=conversation, **kwargs): evaluator
+                    for evaluator in self._evaluators
+                }
+
+                for future in as_completed(futures):
+                    results.update(future.result())
+        else:
+            for evaluator in self._evaluators:
+                result = evaluator(conversation=conversation, **kwargs)
+                results.update(result)
+
+        return results

azure/ai/evaluation/_evaluators/_multimodal/_content_safety_multimodal_base.py

@@ -0,0 +1,55 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+from abc import ABC
+from typing import Union
+from azure.ai.evaluation._common.rai_service import evaluate_with_rai_service_multimodal
+from azure.ai.evaluation._common.constants import EvaluationMetrics, _InternalEvaluationMetrics
+from azure.ai.evaluation._common.utils import validate_conversation
+from azure.core.credentials import TokenCredential
+
+
+class ContentSafetyMultimodalEvaluatorBase(ABC):
+    """
+    Initialize a evaluator for a specified Evaluation Metric. Base class that is not
+    meant to be instantiated by users.
+
+    :param metric: The metric to be evaluated.
+    :type metric: ~azure.ai.evaluation._evaluators._content_safety.flow.constants.EvaluationMetrics
+    :param credential: The credential for connecting to Azure AI project. Required
+    :type credential: ~azure.core.credentials.TokenCredential
+    :param azure_ai_project: The scope of the Azure AI project.
+        It contains subscription id, resource group, and project name.
+    :type azure_ai_project: ~azure.ai.evaluation.AzureAIProject
+    """
+
+    def __init__(
+        self,
+        metric: Union[EvaluationMetrics, _InternalEvaluationMetrics],
+        credential: TokenCredential,
+        azure_ai_project,
+    ):
+        self._metric = metric
+        self._azure_ai_project = azure_ai_project
+        self._credential = credential
+
+    async def __call__(self, *, conversation, **kwargs):
+        """
+        Evaluates content according to this evaluator's metric.
+        :keyword conversation: The conversation contains list of messages to be evaluated.
+            Each message should have "role" and "content" keys.
+        :paramtype conversation: ~azure.ai.evaluation.Conversation
+        :return: The evaluation score computation based on the Content Safety metric (self.metric).
+        :rtype: Dict[str, Union[float, str]]
+        """
+        # validate inputs
+        validate_conversation(conversation)
+        messages = conversation["messages"]
+        # Run score computation based on supplied metric.
+        result = await evaluate_with_rai_service_multimodal(
+            messages=messages,
+            metric_name=self._metric,
+            project_scope=self._azure_ai_project,
+            credential=self._credential,
+        )
+        return result

azure/ai/evaluation/_evaluators/_multimodal/_hate_unfairness.py

@@ -0,0 +1,100 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+from promptflow._utils.async_utils import async_run_allowing_running_loop
+from azure.ai.evaluation._common._experimental import experimental
+from azure.ai.evaluation._common.constants import EvaluationMetrics
+from ._content_safety_multimodal_base import ContentSafetyMultimodalEvaluatorBase
+
+
+@experimental
+class HateUnfairnessMultimodalEvaluator:
+    """
+    Initialize a hate-unfairness multimodal evaluator for hate unfairness score.
+
+    :param credential: The credential for connecting to the Azure AI project. This is required.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :param azure_ai_project: The scope of the Azure AI project,
+        which includes subscription ID, resource group name, and project name.
+    :type azure_ai_project: ~azure.ai.evaluation.AzureAIProject
+
+    **Usage Example**
+
+    .. code-block:: python
+
+        azure_ai_project = {
+            "subscription_id": "<subscription_id>",
+            "resource_group_name": "<resource_group_name>",
+            "project_name": "<project_name>",
+        }
+        eval_fn = HateUnfairnessMultimodalEvaluator(azure_ai_project)
+        result = eval_fn(conversation=
+            {
+                "messages": [
+                    {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "text",
+                                "text": "What's in this image?"
+                            },
+                            {
+                                "type": "image_url",
+                                "image_url": {
+                                    "url": "<image url or base64 encoded image>"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "role": "assistant",
+                        "content": "This picture shows an astronaut standing in the desert."
+                    }
+                ]
+            }
+        )
+
+    **Output Format**
+
+    .. code-block:: json
+
+        {
+            "hate_unfairness": "High",
+            "hate_unfairness_score": 6.5,
+            "hate_unfairness_reason": "Some reason"
+        }
+
+    """
+
+    id = "hate_unfairness_multimodal"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+
+    def __init__(self, credential, azure_ai_project):
+        self._async_evaluator = _AsyncHateUnfairnessMultimodalEvaluator(credential, azure_ai_project)
+
+    def __call__(self, *, conversation, **kwargs):
+        """
+        Evaluates hate unfairness content.
+
+        :keyword conversation: The conversation contains list of messages to be evaluated.
+            Each message should have "role" and "content" keys. It supports single turn only.
+        :paramtype conversation: ~azure.ai.evaluation.Conversation
+        :return: The hate unfairness score.
+        :rtype: Dict[str, Union[float, str]]
+        """
+        return async_run_allowing_running_loop(self._async_evaluator, conversation=conversation, **kwargs)
+
+    def _to_async(self):
+        return self._async_evaluator
+
+
+class _AsyncHateUnfairnessMultimodalEvaluator(ContentSafetyMultimodalEvaluatorBase):
+    def __init__(self, credential, azure_ai_project):
+        super().__init__(
+            metric=EvaluationMetrics.HATE_FAIRNESS,
+            credential=credential,
+            azure_ai_project=azure_ai_project,
+        )
+
+    async def __call__(self, *, conversation, **kwargs):
+        return await super().__call__(conversation=conversation, **kwargs)

azure/ai/evaluation/_evaluators/_multimodal/_protected_material.py

@@ -0,0 +1,124 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+from promptflow._utils.async_utils import async_run_allowing_running_loop
+from azure.ai.evaluation._common.constants import EvaluationMetrics
+from azure.ai.evaluation._common.utils import validate_conversation
+from azure.ai.evaluation._common.rai_service import evaluate_with_rai_service_multimodal
+from azure.ai.evaluation._common._experimental import experimental
+
+
+@experimental
+class ProtectedMaterialMultimodalEvaluator:
+    """
+    Initialize a protected materials evaluator to detect whether protected material
+    is present in multimodal messages. The evaluator outputs a Boolean label (`True` or `False`)
+    indicating the presence of protected material, along with AI-generated reasoning.
+
+    :param credential: The credential for connecting to the Azure AI project. This is required.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :param azure_ai_project: The scope of the Azure AI project, containing the subscription ID,
+        resource group, and project name.
+    :type azure_ai_project: ~azure.ai.evaluation.AzureAIProject
+
+    :return: A dictionary containing the evaluation result label and reasoning.
+
+    **Usage Example**
+
+    .. code-block:: python
+
+        azure_ai_project = {
+            "subscription_id": "<subscription_id>",
+            "resource_group_name": "<resource_group_name>",
+            "project_name": "<project_name>",
+        }
+        eval_fn = ProtectedMaterialMultimodalEvaluator(azure_ai_project)
+        result = eval_fn(conversation=
+            {
+                "messages": [
+                    {
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "text",
+                                "text": "What's in this image?"
+                            },
+                            {
+                                "type": "image_url",
+                                "image_url": {
+                                    "url": "<image url or base64 encoded image>"
+                                }
+                            }
+                        ]
+                    },
+                    {
+                        "role": "assistant",
+                        "content": "This picture shows an astronaut standing in the desert."
+                    }
+                ]
+            }
+        )
+
+    **Output Format**
+
+    .. code-block:: json
+
+        {
+            "protected_material_label": "False",
+            "protected_material_reason": "This query does not contain any protected material."
+        }
+
+    """
+
+    id = "protected_material_multimodal"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+
+    def __init__(
+        self,
+        credential,
+        azure_ai_project,
+    ):
+        self._async_evaluator = _AsyncProtectedMaterialMultimodalEvaluator(credential, azure_ai_project)
+
+    def __call__(self, *, conversation, **kwargs):
+        """
+        Evaluates protected materials content.
+
+        :keyword conversation: The conversation contains list of messages to be evaluated.
+            Each message should have "role" and "content" keys. It supports single turn only.
+        :paramtype conversation: ~azure.ai.evaluation.Conversation
+        :return: A dictionary containing a boolean label and reasoning.
+        :rtype: Dict[str, str]
+        """
+        return async_run_allowing_running_loop(self._async_evaluator, conversation=conversation, **kwargs)
+
+    def _to_async(self):
+        return self._async_evaluator
+
+
+class _AsyncProtectedMaterialMultimodalEvaluator:
+    def __init__(self, credential, azure_ai_project):
+        self._credential = credential
+        self._azure_ai_project = azure_ai_project
+
+    async def __call__(self, *, conversation, **kwargs):
+        """
+        Evaluates content according to this evaluator's metric.
+
+        :keyword conversation: The conversation contains list of messages to be evaluated.
+            Each message should have "role" and "content" keys. It supports single turn only.
+        :paramtype conversation: ~azure.ai.evaluation.Conversation
+        :return: The evaluation score computation based on the Content Safety metric (self.metric).
+        :rtype: Any
+        """
+        # Validate inputs
+        validate_conversation(conversation)
+        messages = conversation["messages"]
+        # Run score computation based on supplied metric.
+        result = await evaluate_with_rai_service_multimodal(
+            messages=messages,
+            metric_name=EvaluationMetrics.PROTECTED_MATERIAL,
+            credential=self._credential,
+            project_scope=self._azure_ai_project,
+        )
+        return result